[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786501898713699-14748-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 02:31:38 dist-test-slave-2w3w krb5kdc[14755](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 02:31:38 dist-test-slave-2w3w krb5kdc[14755](info): set up 2 sockets
Aug 12 02:31:38 dist-test-slave-2w3w krb5kdc[14755](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260812 02:31:39.984171 14748 mini_kdc.cc:121] Time spent starting KDC: real 1.185s user 0.003s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786501898713699-14748-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786501898713699-14748-0/krb5kdc/test-user.keytab.
Aug 12 02:31:40 dist-test-slave-2w3w krb5kdc[14755](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501900, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T02:31:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T02:31:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.103.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.103.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.103.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.62.keytab.
Entry for principal kudu/127.14.103.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.62.keytab.
I20260812 02:31:40.145022 14748 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbep4M4/build/debug/bin/kudu
/tmp/dist-test-taskbep4M4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786501898713699-14748-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786501898713699-14748-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786501898713699-14748-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786501898713699-14748-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.103.62:40639
--webserver_interface=127.14.103.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.62.keytab
--principal=kudu/127.14.103.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.103.20:35379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.103.62:40639
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786501898713699-14748-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786501898713699-14748-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786501898713699-14748-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:31:40.292037 14768 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:31:40.292384 14768 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:31:40.292433 14768 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:31:40.297103 14768 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:31:40.297201 14768 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:31:40.297223 14768 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:31:40.297240 14768 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:31:40.297256 14768 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 02:31:40.304090 14768 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.103.20:35379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786501898713699-14748-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786501898713699-14748-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.103.62:40639
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.62.keytab
--principal=kudu/127.14.103.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.103.62:40639
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786501898713699-14748-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.103.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14768
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786501898713699-14748-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:31:40.305797 14768 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:31:40.307255 14768 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:31:40.322645 14775 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:31:40.323107 14777 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:31:40.323589 14774 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:31:40.324975 14768 server_base.cc:1034] running on GCE node
I20260812 02:31:40.325917 14768 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:31:40.327993 14768 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:31:40.329242 14768 hybrid_clock.cc:648] HybridClock initialized: now 1786501900329197 us; error 67 us; skew 500 ppm
Aug 12 02:31:40 dist-test-slave-2w3w krb5kdc[14755](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501900, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:31:40.333213 14768 init.cc:377] Logged in from keytab as kudu/127.14.103.62@KRBTEST.COM (short username kudu)
I20260812 02:31:40.334648 14768 webserver.cc:460] Webserver started at http://127.14.103.62:40675/ using document root <none> and password file <none>
I20260812 02:31:40.335940 14768 fs_manager.cc:359] Metadata directory not provided
I20260812 02:31:40.336081 14768 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:31:40.336549 14768 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:31:40.340114 14768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786501898713699-14748-0/minicluster-data/master-0/data/instance:
uuid: "bdac89bd540643aca8f4bc8c5f64a10f"
format_stamp: "Formatted at 2026-08-12 02:31:40 on dist-test-slave-2w3w"
server_key: "a82ce68aa9f28635f2a89c0626a9e928"
server_key_iv: "36108975322d84e78f43176f4e3f1afe"
server_key_version: "encryptionkey@0"
I20260812 02:31:40.341223 14768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786501898713699-14748-0/minicluster-data/master-0/wal/instance:
uuid: "bdac89bd540643aca8f4bc8c5f64a10f"
format_stamp: "Formatted at 2026-08-12 02:31:40 on dist-test-slave-2w3w"
server_key: "a82ce68aa9f28635f2a89c0626a9e928"
server_key_iv: "36108975322d84e78f43176f4e3f1afe"
server_key_version: "encryptionkey@0"
I20260812 02:31:40.347389 14768 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260812 02:31:40.351883 14786 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:31:40.354182 14768 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20260812 02:31:40.354498 14768 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786501898713699-14748-0/minicluster-data/master-0/data,/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786501898713699-14748-0/minicluster-data/master-0/wal
uuid: "bdac89bd540643aca8f4bc8c5f64a10f"
format_stamp: "Formatted at 2026-08-12 02:31:40 on dist-test-slave-2w3w"
server_key: "a82ce68aa9f28635f2a89c0626a9e928"
server_key_iv: "36108975322d84e78f43176f4e3f1afe"
server_key_version: "encryptionkey@0"
I20260812 02:31:40.354719 14768 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786501898713699-14748-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786501898713699-14748-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786501898713699-14748-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:31:40.379652 14768 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:31:40.384241 14768 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:31:40.384572 14768 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:31:40.401131 14768 rpc_server.cc:307] RPC server started. Bound to: 127.14.103.62:40639
I20260812 02:31:40.401182 14843 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.103.62:40639 every 8 connection(s)
I20260812 02:31:40.402611 14768 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786501898713699-14748-0/minicluster-data/master-0/data/info.pb
I20260812 02:31:40.406723 14748 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbep4M4/build/debug/bin/kudu as pid 14768
I20260812 02:31:40.407001 14748 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786501898713699-14748-0/minicluster-data/master-0/wal/instance
I20260812 02:31:40.409255 14844 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:31:40.409337 14748 external_mini_cluster.cc:1428] Setting key 8206cca083d8ac1fd882b62c0c83c302
I20260812 02:31:40.422331 14844 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bdac89bd540643aca8f4bc8c5f64a10f: Bootstrap starting.
I20260812 02:31:40.427079 14844 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bdac89bd540643aca8f4bc8c5f64a10f: Neither blocks nor log segments found. Creating new log.
I20260812 02:31:40.428927 14844 log.cc:824] T 00000000000000000000000000000000 P bdac89bd540643aca8f4bc8c5f64a10f: Log is configured to *not* fsync() on all Append() calls
I20260812 02:31:40.434392 14844 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bdac89bd540643aca8f4bc8c5f64a10f: No bootstrap required, opened a new log
I20260812 02:31:40.441534 14844 raft_consensus.cc:348] T 00000000000000000000000000000000 P bdac89bd540643aca8f4bc8c5f64a10f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bdac89bd540643aca8f4bc8c5f64a10f" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 40639 } }
I20260812 02:31:40.442000 14844 raft_consensus.cc:374] T 00000000000000000000000000000000 P bdac89bd540643aca8f4bc8c5f64a10f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:31:40.442126 14844 raft_consensus.cc:729] T 00000000000000000000000000000000 P bdac89bd540643aca8f4bc8c5f64a10f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bdac89bd540643aca8f4bc8c5f64a10f, State: Initialized, Role: FOLLOWER
Aug 12 02:31:40 dist-test-slave-2w3w krb5kdc[14755](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501900, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.103.62@KRBTEST.COM
I20260812 02:31:40.443378 14844 consensus_queue.cc:260] T 00000000000000000000000000000000 P bdac89bd540643aca8f4bc8c5f64a10f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bdac89bd540643aca8f4bc8c5f64a10f" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 40639 } }
I20260812 02:31:40.443670 14844 raft_consensus.cc:388] T 00000000000000000000000000000000 P bdac89bd540643aca8f4bc8c5f64a10f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 02:31:40.443822 14844 raft_consensus.cc:482] T 00000000000000000000000000000000 P bdac89bd540643aca8f4bc8c5f64a10f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 02:31:40.444001 14844 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bdac89bd540643aca8f4bc8c5f64a10f [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:31:40.445829 14844 raft_consensus.cc:504] T 00000000000000000000000000000000 P bdac89bd540643aca8f4bc8c5f64a10f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bdac89bd540643aca8f4bc8c5f64a10f" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 40639 } }
I20260812 02:31:40.446684 14844 leader_election.cc:302] T 00000000000000000000000000000000 P bdac89bd540643aca8f4bc8c5f64a10f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bdac89bd540643aca8f4bc8c5f64a10f; no voters:
I20260812 02:31:40.447274 14844 leader_election.cc:288] T 00000000000000000000000000000000 P bdac89bd540643aca8f4bc8c5f64a10f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 02:31:40.447541 14849 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bdac89bd540643aca8f4bc8c5f64a10f [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:31:40.447783 14849 raft_consensus.cc:686] T 00000000000000000000000000000000 P bdac89bd540643aca8f4bc8c5f64a10f [term 1 LEADER]: Becoming Leader. State: Replica: bdac89bd540643aca8f4bc8c5f64a10f, State: Running, Role: LEADER
I20260812 02:31:40.448312 14849 consensus_queue.cc:237] T 00000000000000000000000000000000 P bdac89bd540643aca8f4bc8c5f64a10f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bdac89bd540643aca8f4bc8c5f64a10f" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 40639 } }
I20260812 02:31:40.449385 14844 sys_catalog.cc:564] T 00000000000000000000000000000000 P bdac89bd540643aca8f4bc8c5f64a10f [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:31:40.450196 14849 sys_catalog.cc:455] T 00000000000000000000000000000000 P bdac89bd540643aca8f4bc8c5f64a10f [sys.catalog]: SysCatalogTable state changed. Reason: New leader bdac89bd540643aca8f4bc8c5f64a10f. Latest consensus state: current_term: 1 leader_uuid: "bdac89bd540643aca8f4bc8c5f64a10f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bdac89bd540643aca8f4bc8c5f64a10f" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 40639 } } }
I20260812 02:31:40.450352 14849 sys_catalog.cc:458] T 00000000000000000000000000000000 P bdac89bd540643aca8f4bc8c5f64a10f [sys.catalog]: This master's current role is: LEADER
I20260812 02:31:40.451138 14855 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 02:31:40.451468 14850 sys_catalog.cc:455] T 00000000000000000000000000000000 P bdac89bd540643aca8f4bc8c5f64a10f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bdac89bd540643aca8f4bc8c5f64a10f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bdac89bd540643aca8f4bc8c5f64a10f" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 40639 } } }
I20260812 02:31:40.451766 14850 sys_catalog.cc:458] T 00000000000000000000000000000000 P bdac89bd540643aca8f4bc8c5f64a10f [sys.catalog]: This master's current role is: LEADER
I20260812 02:31:40.452878 14846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:31:40.417105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58536 (local address 127.14.103.62:40639)
0812 02:31:40.417773 (+ 668us) server_negotiation.cc:206] Beginning negotiation
0812 02:31:40.417825 (+ 52us) server_negotiation.cc:399] Waiting for connection header
0812 02:31:40.421967 (+ 4142us) server_negotiation.cc:407] Connection header received
0812 02:31:40.426003 (+ 4036us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:31:40.426034 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:31:40.426806 (+ 772us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:31:40.427535 (+ 729us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:31:40.430679 (+ 3144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:31:40.433414 (+ 2735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:31:40.436358 (+ 2944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:31:40.437781 (+ 1423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:31:40.438049 (+ 268us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:31:40.443677 (+ 5628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:31:40.443729 (+ 52us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:31:40.443750 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:31:40.443818 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:31:40.446905 (+ 3087us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:31:40.447894 (+ 989us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:31:40.447902 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:31:40.447908 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:31:40.448129 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:31:40.448663 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:31:40.448675 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:31:40.448677 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:31:40.449456 (+ 779us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:31:40.449787 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0812 02:31:40.450888 (+ 1101us) server_negotiation.cc:299] Negotiation successful
0812 02:31:40.451381 (+ 493us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":415,"thread_start_us":165,"threads_started":1}
I20260812 02:31:40.457823 14855 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 02:31:40.469861 14855 catalog_manager.cc:1347] Generated new cluster ID: 2e32832e995541338d15f5e34a8868b4
I20260812 02:31:40.470083 14855 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 02:31:40.493682 14855 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 02:31:40.495822 14855 catalog_manager.cc:1504] Loading token signing keys...
I20260812 02:31:40.509955 14855 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bdac89bd540643aca8f4bc8c5f64a10f: Generated new TSK 0
I20260812 02:31:40.511267 14855 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.103.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.103.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.103.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.1.keytab.
Entry for principal kudu/127.14.103.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.1.keytab.
I20260812 02:31:40.556254 14748 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbep4M4/build/debug/bin/kudu
/tmp/dist-test-taskbep4M4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786501898713699-14748-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786501898713699-14748-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786501898713699-14748-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786501898713699-14748-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.103.1:0
--local_ip_for_outbound_sockets=127.14.103.1
--webserver_interface=127.14.103.1
--webserver_port=0
--tserver_master_addrs=127.14.103.62:40639
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.1.keytab
--principal=kudu/127.14.103.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.103.20:35379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786501898713699-14748-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786501898713699-14748-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786501898713699-14748-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:31:40.732467 14872 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:31:40.732801 14872 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:31:40.732851 14872 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:31:40.738162 14872 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:31:40.738286 14872 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:31:40.738360 14872 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.103.1
I20260812 02:31:40.744575 14872 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.103.20:35379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786501898713699-14748-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786501898713699-14748-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.1.keytab
--principal=kudu/127.14.103.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.103.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786501898713699-14748-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.103.1
--webserver_port=0
--tserver_master_addrs=127.14.103.62:40639
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14872
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.103.1
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786501898713699-14748-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:31:40.745829 14872 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:31:40.746949 14872 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:31:40.759256 14877 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:31:40.759574 14872 server_base.cc:1034] running on GCE node
W20260812 02:31:40.759351 14880 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:31:40.759224 14878 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:31:40.760195 14872 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:31:40.760921 14872 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:31:40.762168 14872 hybrid_clock.cc:648] HybridClock initialized: now 1786501900762138 us; error 33 us; skew 500 ppm
Aug 12 02:31:40 dist-test-slave-2w3w krb5kdc[14755](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501900, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:31:40.766794 14872 init.cc:377] Logged in from keytab as kudu/127.14.103.1@KRBTEST.COM (short username kudu)
I20260812 02:31:40.768049 14872 webserver.cc:460] Webserver started at http://127.14.103.1:43629/ using document root <none> and password file <none>
I20260812 02:31:40.768740 14872 fs_manager.cc:359] Metadata directory not provided
I20260812 02:31:40.768850 14872 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:31:40.769243 14872 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:31:40.772225 14872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786501898713699-14748-0/minicluster-data/ts-0/data/instance:
uuid: "67183879c9f2436b8532a0966f057dfe"
format_stamp: "Formatted at 2026-08-12 02:31:40 on dist-test-slave-2w3w"
server_key: "9f72afc4a58be602d42b7bdae51b7cb5"
server_key_iv: "a18187b938fe43482bc662eef3a6a3a7"
server_key_version: "encryptionkey@0"
I20260812 02:31:40.773123 14872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786501898713699-14748-0/minicluster-data/ts-0/wal/instance:
uuid: "67183879c9f2436b8532a0966f057dfe"
format_stamp: "Formatted at 2026-08-12 02:31:40 on dist-test-slave-2w3w"
server_key: "9f72afc4a58be602d42b7bdae51b7cb5"
server_key_iv: "a18187b938fe43482bc662eef3a6a3a7"
server_key_version: "encryptionkey@0"
I20260812 02:31:40.780557 14872 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.004s sys 0.004s
I20260812 02:31:40.784636 14888 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:31:40.786593 14872 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260812 02:31:40.786850 14872 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786501898713699-14748-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786501898713699-14748-0/minicluster-data/ts-0/wal
uuid: "67183879c9f2436b8532a0966f057dfe"
format_stamp: "Formatted at 2026-08-12 02:31:40 on dist-test-slave-2w3w"
server_key: "9f72afc4a58be602d42b7bdae51b7cb5"
server_key_iv: "a18187b938fe43482bc662eef3a6a3a7"
server_key_version: "encryptionkey@0"
I20260812 02:31:40.787088 14872 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786501898713699-14748-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786501898713699-14748-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786501898713699-14748-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:31:40.815621 14872 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:31:40.821089 14872 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:31:40.821381 14872 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:31:40.822306 14872 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:31:40.823740 14872 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:31:40.823822 14872 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:31:40.823902 14872 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:31:40.823954 14872 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:31:40.847996 14872 rpc_server.cc:307] RPC server started. Bound to: 127.14.103.1:35263
I20260812 02:31:40.848107 15014 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.103.1:35263 every 8 connection(s)
I20260812 02:31:40.849164 14872 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786501898713699-14748-0/minicluster-data/ts-0/data/info.pb
I20260812 02:31:40.858337 14748 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbep4M4/build/debug/bin/kudu as pid 14872
I20260812 02:31:40.858532 14748 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786501898713699-14748-0/minicluster-data/ts-0/wal/instance
I20260812 02:31:40.858866 14748 external_mini_cluster.cc:1428] Setting key b55885ee8fa1cc28fe0151f0cf31569f
Aug 12 02:31:40 dist-test-slave-2w3w krb5kdc[14755](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501900, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.1@KRBTEST.COM for kudu/127.14.103.62@KRBTEST.COM
I20260812 02:31:40.869994 14846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:31:40.852041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.1:47683 (local address 127.14.103.62:40639)
0812 02:31:40.852180 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0812 02:31:40.852185 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:31:40.853897 (+ 1712us) server_negotiation.cc:407] Connection header received
0812 02:31:40.855269 (+ 1372us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:31:40.855273 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:31:40.855336 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:31:40.855413 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:31:40.857620 (+ 2207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:31:40.858193 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:31:40.860512 (+ 2319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:31:40.860861 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:31:40.861008 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:31:40.865650 (+ 4642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:31:40.865657 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:31:40.865666 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:31:40.865698 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:31:40.867465 (+ 1767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:31:40.868158 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:31:40.868161 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:31:40.868163 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:31:40.868247 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:31:40.868682 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:31:40.868685 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:31:40.868687 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:31:40.868911 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:31:40.869043 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0812 02:31:40.869640 (+ 597us) server_negotiation.cc:299] Negotiation successful
0812 02:31:40.869793 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260812 02:31:40.871044 15017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:31:40.852550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:40639 (local address 127.14.103.1:47683)
0812 02:31:40.853644 (+ 1094us) negotiation.cc:107] Waiting for socket to connect
0812 02:31:40.853700 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0812 02:31:40.854878 (+ 1178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:31:40.855663 (+ 785us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:31:40.855672 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:31:40.856498 (+ 826us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:31:40.857400 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:31:40.857423 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:31:40.858369 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:31:40.858377 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:31:40.860351 (+ 1974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:31:40.860366 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:31:40.861285 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:31:40.861296 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:31:40.861730 (+ 434us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:31:40.862764 (+ 1034us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:31:40.862804 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:31:40.865388 (+ 2584us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:31:40.867631 (+ 2243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:31:40.867639 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:31:40.867662 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:31:40.868010 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:31:40.868353 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:31:40.868356 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:31:40.868358 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:31:40.868558 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:31:40.869063 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:31:40.869071 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:31:40.869363 (+ 292us) client_negotiation.cc:769] Sending connection context
0812 02:31:40.869582 (+ 219us) client_negotiation.cc:240] Negotiation successful
0812 02:31:40.869962 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":833,"thread_start_us":232,"threads_started":1}
I20260812 02:31:40.872781 15015 heartbeater.cc:347] Connected to a master server at 127.14.103.62:40639
I20260812 02:31:40.873150 15015 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.14.103.2@KRBTEST.COM; defaulting to no policy
I20260812 02:31:40.873950 15015 heartbeater.cc:511] Master 127.14.103.62:40639 requested a full tablet report, sending...
I20260812 02:31:40.876804 14804 ts_manager.cc:194] Registered new tserver with Master: 67183879c9f2436b8532a0966f057dfe (127.14.103.1:35263)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.103.2@KRBTEST.COM" created.
I20260812 02:31:40.878639 14804 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.103.1@KRBTEST.COM'} at 127.14.103.1:47683
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.103.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.2.keytab.
Entry for principal kudu/127.14.103.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.2.keytab.
I20260812 02:31:40.905180 14748 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbep4M4/build/debug/bin/kudu
/tmp/dist-test-taskbep4M4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786501898713699-14748-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786501898713699-14748-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786501898713699-14748-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786501898713699-14748-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.103.2:0
--local_ip_for_outbound_sockets=127.14.103.2
--webserver_interface=127.14.103.2
--webserver_port=0
--tserver_master_addrs=127.14.103.62:40639
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.2.keytab
--principal=kudu/127.14.103.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.103.20:35379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786501898713699-14748-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786501898713699-14748-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786501898713699-14748-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:31:41.068735 15021 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:31:41.069072 15021 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:31:41.069123 15021 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:31:41.074326 15021 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:31:41.074411 15021 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:31:41.074481 15021 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.103.2
I20260812 02:31:41.080615 15021 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.103.20:35379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786501898713699-14748-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786501898713699-14748-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.2.keytab
--principal=kudu/127.14.103.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.103.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786501898713699-14748-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.103.2
--webserver_port=0
--tserver_master_addrs=127.14.103.62:40639
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15021
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.103.2
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786501898713699-14748-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:31:41.081847 15021 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:31:41.083164 15021 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:31:41.095005 15029 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:31:41.095028 15027 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:31:41.095005 15026 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:31:41.095259 15021 server_base.cc:1034] running on GCE node
I20260812 02:31:41.095954 15021 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:31:41.096750 15021 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:31:41.098038 15021 hybrid_clock.cc:648] HybridClock initialized: now 1786501901097991 us; error 65 us; skew 500 ppm
Aug 12 02:31:41 dist-test-slave-2w3w krb5kdc[14755](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501901, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:31:41.101222 15021 init.cc:377] Logged in from keytab as kudu/127.14.103.2@KRBTEST.COM (short username kudu)
I20260812 02:31:41.102272 15021 webserver.cc:460] Webserver started at http://127.14.103.2:34775/ using document root <none> and password file <none>
I20260812 02:31:41.102833 15021 fs_manager.cc:359] Metadata directory not provided
I20260812 02:31:41.102895 15021 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:31:41.103144 15021 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:31:41.105480 15021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786501898713699-14748-0/minicluster-data/ts-1/data/instance:
uuid: "d9a216c6c3c447a881b9a4a78cbf7dae"
format_stamp: "Formatted at 2026-08-12 02:31:41 on dist-test-slave-2w3w"
server_key: "f4a1fafced8f47edcd48a80397c31fd3"
server_key_iv: "ade0459081e6992a2f607c18d7edec8e"
server_key_version: "encryptionkey@0"
I20260812 02:31:41.106164 15021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786501898713699-14748-0/minicluster-data/ts-1/wal/instance:
uuid: "d9a216c6c3c447a881b9a4a78cbf7dae"
format_stamp: "Formatted at 2026-08-12 02:31:41 on dist-test-slave-2w3w"
server_key: "f4a1fafced8f47edcd48a80397c31fd3"
server_key_iv: "ade0459081e6992a2f607c18d7edec8e"
server_key_version: "encryptionkey@0"
I20260812 02:31:41.111135 15021 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260812 02:31:41.114019 15038 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:31:41.115994 15021 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 02:31:41.116217 15021 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786501898713699-14748-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786501898713699-14748-0/minicluster-data/ts-1/wal
uuid: "d9a216c6c3c447a881b9a4a78cbf7dae"
format_stamp: "Formatted at 2026-08-12 02:31:41 on dist-test-slave-2w3w"
server_key: "f4a1fafced8f47edcd48a80397c31fd3"
server_key_iv: "ade0459081e6992a2f607c18d7edec8e"
server_key_version: "encryptionkey@0"
I20260812 02:31:41.116456 15021 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786501898713699-14748-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786501898713699-14748-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786501898713699-14748-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:31:41.136222 15021 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:31:41.140689 15021 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:31:41.140945 15021 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:31:41.141681 15021 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:31:41.142712 15021 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:31:41.142766 15021 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:31:41.142809 15021 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:31:41.142863 15021 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:31:41.155527 15021 rpc_server.cc:307] RPC server started. Bound to: 127.14.103.2:35385
I20260812 02:31:41.155570 15164 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.103.2:35385 every 8 connection(s)
I20260812 02:31:41.156582 15021 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786501898713699-14748-0/minicluster-data/ts-1/data/info.pb
I20260812 02:31:41.166129 14748 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbep4M4/build/debug/bin/kudu as pid 15021
I20260812 02:31:41.166231 14748 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786501898713699-14748-0/minicluster-data/ts-1/wal/instance
I20260812 02:31:41.166515 14748 external_mini_cluster.cc:1428] Setting key de8bd0d6c7a56dc7e7628229bde935f9
Aug 12 02:31:41 dist-test-slave-2w3w krb5kdc[14755](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501901, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.2@KRBTEST.COM for kudu/127.14.103.62@KRBTEST.COM
I20260812 02:31:41.177281 14846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:31:41.159068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.2:44211 (local address 127.14.103.62:40639)
0812 02:31:41.159239 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0812 02:31:41.159245 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 02:31:41.160897 (+ 1652us) server_negotiation.cc:407] Connection header received
0812 02:31:41.162451 (+ 1554us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:31:41.162455 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:31:41.162512 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:31:41.162598 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:31:41.164203 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:31:41.164932 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:31:41.167615 (+ 2683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:31:41.167856 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:31:41.167991 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:31:41.171365 (+ 3374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:31:41.171373 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:31:41.171379 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:31:41.171429 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:31:41.174010 (+ 2581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:31:41.175003 (+ 993us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:31:41.175011 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:31:41.175016 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:31:41.175145 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:31:41.175684 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:31:41.175691 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:31:41.175695 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:31:41.176014 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:31:41.176146 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0812 02:31:41.176805 (+ 659us) server_negotiation.cc:299] Negotiation successful
0812 02:31:41.176995 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260812 02:31:41.178009 15169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:31:41.159575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:40639 (local address 127.14.103.2:44211)
0812 02:31:41.160693 (+ 1118us) negotiation.cc:107] Waiting for socket to connect
0812 02:31:41.160744 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0812 02:31:41.162005 (+ 1261us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:31:41.162846 (+ 841us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:31:41.162855 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:31:41.163341 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:31:41.163980 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:31:41.163998 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:31:41.165302 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:31:41.165306 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:31:41.167444 (+ 2138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:31:41.167467 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:31:41.168038 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:31:41.168042 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:31:41.168276 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:31:41.169310 (+ 1034us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:31:41.169331 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:31:41.171100 (+ 1769us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:31:41.174328 (+ 3228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:31:41.174335 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:31:41.174352 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:31:41.174706 (+ 354us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:31:41.175343 (+ 637us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:31:41.175345 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:31:41.175347 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:31:41.175538 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:31:41.176172 (+ 634us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:31:41.176186 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:31:41.176499 (+ 313us) client_negotiation.cc:769] Sending connection context
0812 02:31:41.176677 (+ 178us) client_negotiation.cc:240] Negotiation successful
0812 02:31:41.177011 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":754,"thread_start_us":204,"threads_started":1}
I20260812 02:31:41.179510 15166 heartbeater.cc:347] Connected to a master server at 127.14.103.62:40639
I20260812 02:31:41.179821 15166 heartbeater.cc:464] Registering TS with master...
I20260812 02:31:41.180550 15166 heartbeater.cc:511] Master 127.14.103.62:40639 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.14.103.3@KRBTEST.COM; defaulting to no policy
I20260812 02:31:41.181945 14804 ts_manager.cc:194] Registered new tserver with Master: d9a216c6c3c447a881b9a4a78cbf7dae (127.14.103.2:35385)
I20260812 02:31:41.182691 14804 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.103.2@KRBTEST.COM'} at 127.14.103.2:44211
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.103.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.103.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.3.keytab.
Entry for principal kudu/127.14.103.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.3.keytab.
I20260812 02:31:41.212360 14748 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbep4M4/build/debug/bin/kudu
/tmp/dist-test-taskbep4M4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786501898713699-14748-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786501898713699-14748-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786501898713699-14748-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786501898713699-14748-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.103.3:0
--local_ip_for_outbound_sockets=127.14.103.3
--webserver_interface=127.14.103.3
--webserver_port=0
--tserver_master_addrs=127.14.103.62:40639
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.3.keytab
--principal=kudu/127.14.103.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.103.20:35379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786501898713699-14748-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786501898713699-14748-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786501898713699-14748-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:31:41.346499 15172 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:31:41.346839 15172 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:31:41.346890 15172 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:31:41.352380 15172 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:31:41.352502 15172 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:31:41.352593 15172 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.103.3
I20260812 02:31:41.359078 15172 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.103.20:35379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786501898713699-14748-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786501898713699-14748-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.3.keytab
--principal=kudu/127.14.103.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.103.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786501898713699-14748-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.103.3
--webserver_port=0
--tserver_master_addrs=127.14.103.62:40639
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15172
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.103.3
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786501898713699-14748-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:31:41.360306 15172 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:31:41.361409 15172 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:31:41.371973 15181 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:31:41.371994 15179 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:31:41.371974 15177 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:31:41.372268 15172 server_base.cc:1034] running on GCE node
I20260812 02:31:41.372843 15172 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:31:41.373605 15172 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:31:41.374807 15172 hybrid_clock.cc:648] HybridClock initialized: now 1786501901374781 us; error 43 us; skew 500 ppm
Aug 12 02:31:41 dist-test-slave-2w3w krb5kdc[14755](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501901, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:31:41.378683 15172 init.cc:377] Logged in from keytab as kudu/127.14.103.3@KRBTEST.COM (short username kudu)
I20260812 02:31:41.379843 15172 webserver.cc:460] Webserver started at http://127.14.103.3:43235/ using document root <none> and password file <none>
I20260812 02:31:41.380499 15172 fs_manager.cc:359] Metadata directory not provided
I20260812 02:31:41.380595 15172 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:31:41.380837 15172 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:31:41.383332 15172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786501898713699-14748-0/minicluster-data/ts-2/data/instance:
uuid: "43a956d938c142aab3f0c152ab0b1fce"
format_stamp: "Formatted at 2026-08-12 02:31:41 on dist-test-slave-2w3w"
server_key: "9a0bcc267630f1e2c900ba3474de8fae"
server_key_iv: "a544959f6c9046ee97d1ce441353c969"
server_key_version: "encryptionkey@0"
I20260812 02:31:41.384054 15172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786501898713699-14748-0/minicluster-data/ts-2/wal/instance:
uuid: "43a956d938c142aab3f0c152ab0b1fce"
format_stamp: "Formatted at 2026-08-12 02:31:41 on dist-test-slave-2w3w"
server_key: "9a0bcc267630f1e2c900ba3474de8fae"
server_key_iv: "a544959f6c9046ee97d1ce441353c969"
server_key_version: "encryptionkey@0"
I20260812 02:31:41.388957 15172 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260812 02:31:41.391752 15188 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:31:41.393291 15172 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 02:31:41.393466 15172 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786501898713699-14748-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786501898713699-14748-0/minicluster-data/ts-2/wal
uuid: "43a956d938c142aab3f0c152ab0b1fce"
format_stamp: "Formatted at 2026-08-12 02:31:41 on dist-test-slave-2w3w"
server_key: "9a0bcc267630f1e2c900ba3474de8fae"
server_key_iv: "a544959f6c9046ee97d1ce441353c969"
server_key_version: "encryptionkey@0"
I20260812 02:31:41.393601 15172 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786501898713699-14748-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786501898713699-14748-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786501898713699-14748-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:31:41.424222 15172 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:31:41.429241 15172 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:31:41.429495 15172 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:31:41.430209 15172 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:31:41.431360 15172 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:31:41.431417 15172 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:31:41.431463 15172 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:31:41.431514 15172 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:31:41.446218 15172 rpc_server.cc:307] RPC server started. Bound to: 127.14.103.3:38387
I20260812 02:31:41.446237 15311 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.103.3:38387 every 8 connection(s)
I20260812 02:31:41.447718 15172 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786501898713699-14748-0/minicluster-data/ts-2/data/info.pb
I20260812 02:31:41.452391 14748 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbep4M4/build/debug/bin/kudu as pid 15172
I20260812 02:31:41.452556 14748 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786501898713699-14748-0/minicluster-data/ts-2/wal/instance
I20260812 02:31:41.452844 14748 external_mini_cluster.cc:1428] Setting key b021e60c5c1adbc8e32a901e5ef4a584
Aug 12 02:31:41 dist-test-slave-2w3w krb5kdc[14755](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501901, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.3@KRBTEST.COM for kudu/127.14.103.62@KRBTEST.COM
I20260812 02:31:41.469681 14846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:31:41.450324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.3:54271 (local address 127.14.103.62:40639)
0812 02:31:41.450753 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0812 02:31:41.450757 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:31:41.451870 (+ 1113us) server_negotiation.cc:407] Connection header received
0812 02:31:41.454404 (+ 2534us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:31:41.454407 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:31:41.454504 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:31:41.454584 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:31:41.457463 (+ 2879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:31:41.458030 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:31:41.460615 (+ 2585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:31:41.460920 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:31:41.461230 (+ 310us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:31:41.465094 (+ 3864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:31:41.465097 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:31:41.465100 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:31:41.465129 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:31:41.466840 (+ 1711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:31:41.467656 (+ 816us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:31:41.467660 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:31:41.467661 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:31:41.467775 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:31:41.468179 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:31:41.468183 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:31:41.468185 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:31:41.468448 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:31:41.468556 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0812 02:31:41.469305 (+ 749us) server_negotiation.cc:299] Negotiation successful
0812 02:31:41.469490 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143}
I20260812 02:31:41.470582 15315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:31:41.450773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:40639 (local address 127.14.103.3:54271)
0812 02:31:41.451647 (+ 874us) negotiation.cc:107] Waiting for socket to connect
0812 02:31:41.451699 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0812 02:31:41.453839 (+ 2140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:31:41.454808 (+ 969us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:31:41.454819 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:31:41.455813 (+ 994us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:31:41.457089 (+ 1276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:31:41.457254 (+ 165us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:31:41.458343 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:31:41.458349 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:31:41.460226 (+ 1877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:31:41.460254 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:31:41.461367 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:31:41.461381 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:31:41.461731 (+ 350us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:31:41.462533 (+ 802us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:31:41.462560 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:31:41.464845 (+ 2285us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:31:41.467035 (+ 2190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:31:41.467043 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:31:41.467069 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:31:41.467435 (+ 366us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:31:41.467882 (+ 447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:31:41.467885 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:31:41.467892 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:31:41.468065 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:31:41.468633 (+ 568us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:31:41.468639 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:31:41.469024 (+ 385us) client_negotiation.cc:769] Sending connection context
0812 02:31:41.469206 (+ 182us) client_negotiation.cc:240] Negotiation successful
0812 02:31:41.469601 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":632,"thread_start_us":288,"threads_started":1}
I20260812 02:31:41.472404 15312 heartbeater.cc:347] Connected to a master server at 127.14.103.62:40639
I20260812 02:31:41.472720 15312 heartbeater.cc:464] Registering TS with master...
I20260812 02:31:41.473312 15312 heartbeater.cc:511] Master 127.14.103.62:40639 requested a full tablet report, sending...
I20260812 02:31:41.474653 14804 ts_manager.cc:194] Registered new tserver with Master: 43a956d938c142aab3f0c152ab0b1fce (127.14.103.3:38387)
I20260812 02:31:41.475364 14804 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.103.3@KRBTEST.COM'} at 127.14.103.3:54271
I20260812 02:31:41.482537 14748 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 12 02:31:41 dist-test-slave-2w3w krb5kdc[14755](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501901, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 12 02:31:41 dist-test-slave-2w3w krb5kdc[14755](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501901, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.14.103.1@KRBTEST.COM
I20260812 02:31:41.515851 15320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:31:41.504634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38486 (local address 127.14.103.1:35263)
0812 02:31:41.504932 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0812 02:31:41.504942 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 02:31:41.505014 (+ 72us) server_negotiation.cc:407] Connection header received
0812 02:31:41.505210 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:31:41.505215 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:31:41.505433 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:31:41.505590 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:31:41.506377 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:31:41.507257 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:31:41.508167 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:31:41.508583 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:31:41.508708 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:31:41.511698 (+ 2990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:31:41.511713 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:31:41.511729 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:31:41.511765 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:31:41.513741 (+ 1976us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:31:41.514278 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:31:41.514284 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:31:41.514285 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:31:41.514402 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:31:41.514692 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:31:41.514697 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:31:41.514699 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:31:41.515053 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:31:41.515282 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0812 02:31:41.515489 (+ 207us) server_negotiation.cc:299] Negotiation successful
0812 02:31:41.515672 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":69,"threads_started":1}
W20260812 02:31:41.516990 14918 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38486
Aug 12 02:31:41 dist-test-slave-2w3w krb5kdc[14755](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501901, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 12 02:31:41 dist-test-slave-2w3w krb5kdc[14755](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501901, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.103.1@KRBTEST.COM
I20260812 02:31:41.549219 15320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:31:41.539517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38496 (local address 127.14.103.1:35263)
0812 02:31:41.539774 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0812 02:31:41.539779 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:31:41.539956 (+ 177us) server_negotiation.cc:407] Connection header received
0812 02:31:41.540295 (+ 339us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:31:41.540299 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:31:41.540365 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:31:41.540459 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:31:41.541293 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:31:41.541812 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:31:41.542686 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:31:41.542922 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:31:41.543096 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:31:41.545692 (+ 2596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:31:41.545695 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:31:41.545697 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:31:41.545727 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:31:41.547494 (+ 1767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:31:41.547948 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:31:41.547954 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:31:41.547957 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:31:41.548042 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:31:41.548317 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:31:41.548321 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:31:41.548323 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:31:41.548522 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:31:41.548668 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0812 02:31:41.548909 (+ 241us) server_negotiation.cc:299] Negotiation successful
0812 02:31:41.549043 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260812 02:31:41.550391 14748 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbep4M4/build/debug/bin/kudu with pid 14872
I20260812 02:31:41.560500 14748 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbep4M4/build/debug/bin/kudu with pid 15021
I20260812 02:31:41.568238 14748 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbep4M4/build/debug/bin/kudu with pid 15172
I20260812 02:31:41.577559 14748 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbep4M4/build/debug/bin/kudu with pid 14768
2026-08-12T02:31:41Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2821 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786501898713699-14748-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 02:31:41 dist-test-slave-2w3w krb5kdc[15328](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 02:31:41 dist-test-slave-2w3w krb5kdc[15328](info): set up 2 sockets
Aug 12 02:31:41 dist-test-slave-2w3w krb5kdc[15328](info): commencing operation
krb5kdc: starting...
W20260812 02:31:42.839020 14748 mini_kdc.cc:121] Time spent starting KDC: real 1.233s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786501898713699-14748-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786501898713699-14748-0/krb5kdc/test-user.keytab.
Aug 12 02:31:42 dist-test-slave-2w3w krb5kdc[15328](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501902, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T02:31:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T02:31:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.103.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.103.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.103.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.62.keytab.
Entry for principal kudu/127.14.103.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.62.keytab.
I20260812 02:31:42.991943 14748 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbep4M4/build/debug/bin/kudu
/tmp/dist-test-taskbep4M4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786501898713699-14748-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786501898713699-14748-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786501898713699-14748-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786501898713699-14748-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.103.62:46683
--webserver_interface=127.14.103.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.62.keytab
--principal=kudu/127.14.103.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.103.20:40723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.103.62:46683
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786501898713699-14748-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786501898713699-14748-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786501898713699-14748-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:31:43.124447 15343 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:31:43.124771 15343 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:31:43.124814 15343 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:31:43.129031 15343 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:31:43.129112 15343 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:31:43.129179 15343 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:31:43.129205 15343 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:31:43.129230 15343 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 02:31:43.135072 15343 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.103.20:40723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786501898713699-14748-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786501898713699-14748-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.103.62:46683
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.62.keytab
--principal=kudu/127.14.103.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.103.62:46683
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786501898713699-14748-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.103.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15343
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786501898713699-14748-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:31:43.136260 15343 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:31:43.137326 15343 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:31:43.146690 15351 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:31:43.146823 15348 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:31:43.146914 15343 server_base.cc:1034] running on GCE node
W20260812 02:31:43.146690 15349 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:31:43.147818 15343 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:31:43.149087 15343 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:31:43.150305 15343 hybrid_clock.cc:648] HybridClock initialized: now 1786501903150268 us; error 56 us; skew 500 ppm
Aug 12 02:31:43 dist-test-slave-2w3w krb5kdc[15328](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501903, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:31:43.157828 15343 init.cc:377] Logged in from keytab as kudu/127.14.103.62@KRBTEST.COM (short username kudu)
I20260812 02:31:43.159296 15343 webserver.cc:460] Webserver started at http://127.14.103.62:39045/ using document root <none> and password file <none>
I20260812 02:31:43.159971 15343 fs_manager.cc:359] Metadata directory not provided
I20260812 02:31:43.160079 15343 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:31:43.160543 15343 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:31:43.163039 15343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786501898713699-14748-0/minicluster-data/master-0/data/instance:
uuid: "29ffe624bb42463eaa9a6573d0b2148a"
format_stamp: "Formatted at 2026-08-12 02:31:43 on dist-test-slave-2w3w"
server_key: "18cc558bd5ff8189c50cd78c47a2f801"
server_key_iv: "49a6d0474603dc63f0288e4f024e6714"
server_key_version: "encryptionkey@0"
I20260812 02:31:43.163726 15343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786501898713699-14748-0/minicluster-data/master-0/wal/instance:
uuid: "29ffe624bb42463eaa9a6573d0b2148a"
format_stamp: "Formatted at 2026-08-12 02:31:43 on dist-test-slave-2w3w"
server_key: "18cc558bd5ff8189c50cd78c47a2f801"
server_key_iv: "49a6d0474603dc63f0288e4f024e6714"
server_key_version: "encryptionkey@0"
I20260812 02:31:43.168365 15343 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260812 02:31:43.171414 15358 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:31:43.172829 15343 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260812 02:31:43.173035 15343 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786501898713699-14748-0/minicluster-data/master-0/data,/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786501898713699-14748-0/minicluster-data/master-0/wal
uuid: "29ffe624bb42463eaa9a6573d0b2148a"
format_stamp: "Formatted at 2026-08-12 02:31:43 on dist-test-slave-2w3w"
server_key: "18cc558bd5ff8189c50cd78c47a2f801"
server_key_iv: "49a6d0474603dc63f0288e4f024e6714"
server_key_version: "encryptionkey@0"
I20260812 02:31:43.173187 15343 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786501898713699-14748-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786501898713699-14748-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786501898713699-14748-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:31:43.206655 15343 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:31:43.210430 15343 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:31:43.210691 15343 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:31:43.219092 15343 rpc_server.cc:307] RPC server started. Bound to: 127.14.103.62:46683
I20260812 02:31:43.219123 15416 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.103.62:46683 every 8 connection(s)
I20260812 02:31:43.220167 15343 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786501898713699-14748-0/minicluster-data/master-0/data/info.pb
I20260812 02:31:43.221189 14748 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbep4M4/build/debug/bin/kudu as pid 15343
I20260812 02:31:43.221305 14748 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786501898713699-14748-0/minicluster-data/master-0/wal/instance
I20260812 02:31:43.221621 14748 external_mini_cluster.cc:1428] Setting key 32e67fa1ffd5aba3ef26fda66d88d22b
I20260812 02:31:43.223898 15417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:31:43.229998 15417 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29ffe624bb42463eaa9a6573d0b2148a: Bootstrap starting.
Aug 12 02:31:43 dist-test-slave-2w3w krb5kdc[15328](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501902, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.103.62@KRBTEST.COM
I20260812 02:31:43.232777 15417 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 29ffe624bb42463eaa9a6573d0b2148a: Neither blocks nor log segments found. Creating new log.
I20260812 02:31:43.233647 15417 log.cc:824] T 00000000000000000000000000000000 P 29ffe624bb42463eaa9a6573d0b2148a: Log is configured to *not* fsync() on all Append() calls
I20260812 02:31:43.235865 15417 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29ffe624bb42463eaa9a6573d0b2148a: No bootstrap required, opened a new log
I20260812 02:31:43.237217 15420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:31:43.223135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49150 (local address 127.14.103.62:46683)
0812 02:31:43.223643 (+ 508us) server_negotiation.cc:206] Beginning negotiation
0812 02:31:43.223659 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0812 02:31:43.223696 (+ 37us) server_negotiation.cc:407] Connection header received
0812 02:31:43.224446 (+ 750us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:31:43.224465 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:31:43.224800 (+ 335us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:31:43.225169 (+ 369us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:31:43.226094 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:31:43.227214 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:31:43.228118 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:31:43.228628 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:31:43.228816 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:31:43.231752 (+ 2936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:31:43.231757 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:31:43.231773 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:31:43.231809 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:31:43.233813 (+ 2004us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:31:43.234390 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:31:43.234396 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:31:43.234402 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:31:43.234537 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:31:43.234862 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:31:43.234865 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:31:43.234866 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:31:43.235372 (+ 506us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:31:43.235584 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0812 02:31:43.235887 (+ 303us) server_negotiation.cc:299] Negotiation successful
0812 02:31:43.236253 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":178,"threads_started":1}
I20260812 02:31:43.240198 15417 raft_consensus.cc:348] T 00000000000000000000000000000000 P 29ffe624bb42463eaa9a6573d0b2148a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29ffe624bb42463eaa9a6573d0b2148a" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 46683 } }
I20260812 02:31:43.240510 15417 raft_consensus.cc:374] T 00000000000000000000000000000000 P 29ffe624bb42463eaa9a6573d0b2148a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:31:43.240607 15417 raft_consensus.cc:729] T 00000000000000000000000000000000 P 29ffe624bb42463eaa9a6573d0b2148a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29ffe624bb42463eaa9a6573d0b2148a, State: Initialized, Role: FOLLOWER
I20260812 02:31:43.241094 15417 consensus_queue.cc:260] T 00000000000000000000000000000000 P 29ffe624bb42463eaa9a6573d0b2148a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29ffe624bb42463eaa9a6573d0b2148a" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 46683 } }
I20260812 02:31:43.241259 15417 raft_consensus.cc:388] T 00000000000000000000000000000000 P 29ffe624bb42463eaa9a6573d0b2148a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 02:31:43.241353 15417 raft_consensus.cc:482] T 00000000000000000000000000000000 P 29ffe624bb42463eaa9a6573d0b2148a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 02:31:43.241478 15417 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 29ffe624bb42463eaa9a6573d0b2148a [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:31:43.242580 15417 raft_consensus.cc:504] T 00000000000000000000000000000000 P 29ffe624bb42463eaa9a6573d0b2148a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29ffe624bb42463eaa9a6573d0b2148a" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 46683 } }
I20260812 02:31:43.243044 15417 leader_election.cc:302] T 00000000000000000000000000000000 P 29ffe624bb42463eaa9a6573d0b2148a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 29ffe624bb42463eaa9a6573d0b2148a; no voters:
I20260812 02:31:43.243464 15417 leader_election.cc:288] T 00000000000000000000000000000000 P 29ffe624bb42463eaa9a6573d0b2148a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 02:31:43.243566 15422 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 29ffe624bb42463eaa9a6573d0b2148a [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:31:43.243763 15422 raft_consensus.cc:686] T 00000000000000000000000000000000 P 29ffe624bb42463eaa9a6573d0b2148a [term 1 LEADER]: Becoming Leader. State: Replica: 29ffe624bb42463eaa9a6573d0b2148a, State: Running, Role: LEADER
I20260812 02:31:43.244096 15422 consensus_queue.cc:237] T 00000000000000000000000000000000 P 29ffe624bb42463eaa9a6573d0b2148a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29ffe624bb42463eaa9a6573d0b2148a" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 46683 } }
I20260812 02:31:43.244923 15417 sys_catalog.cc:564] T 00000000000000000000000000000000 P 29ffe624bb42463eaa9a6573d0b2148a [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:31:43.245457 15422 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29ffe624bb42463eaa9a6573d0b2148a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 29ffe624bb42463eaa9a6573d0b2148a. Latest consensus state: current_term: 1 leader_uuid: "29ffe624bb42463eaa9a6573d0b2148a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29ffe624bb42463eaa9a6573d0b2148a" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 46683 } } }
I20260812 02:31:43.245586 15422 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29ffe624bb42463eaa9a6573d0b2148a [sys.catalog]: This master's current role is: LEADER
I20260812 02:31:43.245955 15424 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29ffe624bb42463eaa9a6573d0b2148a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "29ffe624bb42463eaa9a6573d0b2148a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29ffe624bb42463eaa9a6573d0b2148a" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 46683 } } }
I20260812 02:31:43.246110 15424 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29ffe624bb42463eaa9a6573d0b2148a [sys.catalog]: This master's current role is: LEADER
I20260812 02:31:43.246055 15427 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 02:31:43.249733 15427 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 02:31:43.256639 15427 catalog_manager.cc:1347] Generated new cluster ID: 44f0b089baa64f539d0ed2150b74f2ae
I20260812 02:31:43.256742 15427 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 02:31:43.287366 15427 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 02:31:43.288414 15427 catalog_manager.cc:1504] Loading token signing keys...
I20260812 02:31:43.296837 15427 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 29ffe624bb42463eaa9a6573d0b2148a: Generated new TSK 0
I20260812 02:31:43.298822 15427 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.103.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.103.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.103.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.1.keytab.
Entry for principal kudu/127.14.103.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.1.keytab.
I20260812 02:31:43.344560 14748 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbep4M4/build/debug/bin/kudu
/tmp/dist-test-taskbep4M4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786501898713699-14748-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786501898713699-14748-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786501898713699-14748-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786501898713699-14748-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.103.1:0
--local_ip_for_outbound_sockets=127.14.103.1
--webserver_interface=127.14.103.1
--webserver_port=0
--tserver_master_addrs=127.14.103.62:46683
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.1.keytab
--principal=kudu/127.14.103.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.103.20:40723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786501898713699-14748-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786501898713699-14748-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786501898713699-14748-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:31:43.474273 15446 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:31:43.474593 15446 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:31:43.474642 15446 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:31:43.479254 15446 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:31:43.479351 15446 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:31:43.479419 15446 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.103.1
I20260812 02:31:43.485435 15446 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.103.20:40723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786501898713699-14748-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786501898713699-14748-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.1.keytab
--principal=kudu/127.14.103.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.103.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786501898713699-14748-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.103.1
--webserver_port=0
--tserver_master_addrs=127.14.103.62:46683
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15446
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.103.1
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786501898713699-14748-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:31:43.486686 15446 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:31:43.487826 15446 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:31:43.498898 15454 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:31:43.498920 15457 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:31:43.499050 15446 server_base.cc:1034] running on GCE node
W20260812 02:31:43.498889 15455 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:31:43.499752 15446 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:31:43.500699 15446 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:31:43.501957 15446 hybrid_clock.cc:648] HybridClock initialized: now 1786501903501924 us; error 56 us; skew 500 ppm
Aug 12 02:31:43 dist-test-slave-2w3w krb5kdc[15328](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501903, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:31:43.506403 15446 init.cc:377] Logged in from keytab as kudu/127.14.103.1@KRBTEST.COM (short username kudu)
I20260812 02:31:43.507653 15446 webserver.cc:460] Webserver started at http://127.14.103.1:44731/ using document root <none> and password file <none>
I20260812 02:31:43.508258 15446 fs_manager.cc:359] Metadata directory not provided
I20260812 02:31:43.508610 15446 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:31:43.508862 15446 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:31:43.511348 15446 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786501898713699-14748-0/minicluster-data/ts-0/data/instance:
uuid: "9feda69f42f44dc98b594f71d2752286"
format_stamp: "Formatted at 2026-08-12 02:31:43 on dist-test-slave-2w3w"
server_key: "fbf01781006234e26f7f1e21f876e50e"
server_key_iv: "d7117f65434d180bce6c0a1325c0a210"
server_key_version: "encryptionkey@0"
I20260812 02:31:43.512020 15446 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786501898713699-14748-0/minicluster-data/ts-0/wal/instance:
uuid: "9feda69f42f44dc98b594f71d2752286"
format_stamp: "Formatted at 2026-08-12 02:31:43 on dist-test-slave-2w3w"
server_key: "fbf01781006234e26f7f1e21f876e50e"
server_key_iv: "d7117f65434d180bce6c0a1325c0a210"
server_key_version: "encryptionkey@0"
I20260812 02:31:43.516574 15446 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260812 02:31:43.519390 15465 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:31:43.520833 15446 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260812 02:31:43.520977 15446 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786501898713699-14748-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786501898713699-14748-0/minicluster-data/ts-0/wal
uuid: "9feda69f42f44dc98b594f71d2752286"
format_stamp: "Formatted at 2026-08-12 02:31:43 on dist-test-slave-2w3w"
server_key: "fbf01781006234e26f7f1e21f876e50e"
server_key_iv: "d7117f65434d180bce6c0a1325c0a210"
server_key_version: "encryptionkey@0"
I20260812 02:31:43.521095 15446 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786501898713699-14748-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786501898713699-14748-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786501898713699-14748-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:31:43.541769 15446 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:31:43.546228 15446 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:31:43.546416 15446 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:31:43.547130 15446 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:31:43.548027 15446 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:31:43.548079 15446 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:31:43.548118 15446 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:31:43.548141 15446 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:31:43.557763 15446 rpc_server.cc:307] RPC server started. Bound to: 127.14.103.1:35585
I20260812 02:31:43.557787 15588 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.103.1:35585 every 8 connection(s)
I20260812 02:31:43.558848 15446 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786501898713699-14748-0/minicluster-data/ts-0/data/info.pb
I20260812 02:31:43.563917 14748 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbep4M4/build/debug/bin/kudu as pid 15446
I20260812 02:31:43.564020 14748 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786501898713699-14748-0/minicluster-data/ts-0/wal/instance
I20260812 02:31:43.564409 14748 external_mini_cluster.cc:1428] Setting key d1da3dab2a481ec84555340bd25ccf24
Aug 12 02:31:43 dist-test-slave-2w3w krb5kdc[15328](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501903, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.1@KRBTEST.COM for kudu/127.14.103.62@KRBTEST.COM
I20260812 02:31:43.574681 15420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:31:43.560887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.1:60603 (local address 127.14.103.62:46683)
0812 02:31:43.561066 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0812 02:31:43.561069 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 02:31:43.561883 (+ 814us) server_negotiation.cc:407] Connection header received
0812 02:31:43.562877 (+ 994us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:31:43.562880 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:31:43.562942 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:31:43.563077 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:31:43.564685 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:31:43.565423 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:31:43.566759 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:31:43.567064 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:31:43.567181 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:31:43.570263 (+ 3082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:31:43.570267 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:31:43.570269 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:31:43.570296 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:31:43.572139 (+ 1843us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:31:43.572811 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:31:43.572815 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:31:43.572816 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:31:43.572916 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:31:43.573325 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:31:43.573329 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:31:43.573331 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:31:43.573585 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:31:43.573702 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0812 02:31:43.574321 (+ 619us) server_negotiation.cc:299] Negotiation successful
0812 02:31:43.574501 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
WARNING: no policy specified for kudu/127.14.103.2@KRBTEST.COM; defaulting to no policy
I20260812 02:31:43.575731 15591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:31:43.561229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:46683 (local address 127.14.103.1:60603)
0812 02:31:43.561727 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0812 02:31:43.561774 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0812 02:31:43.562634 (+ 860us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:31:43.563435 (+ 801us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:31:43.563444 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:31:43.563873 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:31:43.564518 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:31:43.564536 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:31:43.565571 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:31:43.565575 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:31:43.566621 (+ 1046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:31:43.566632 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:31:43.567237 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:31:43.567241 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:31:43.567426 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:31:43.568111 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:31:43.568129 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:31:43.570029 (+ 1900us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:31:43.572305 (+ 2276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:31:43.572313 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:31:43.572334 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:31:43.572679 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:31:43.573038 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:31:43.573046 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:31:43.573047 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:31:43.573217 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:31:43.573723 (+ 506us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:31:43.573731 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:31:43.574048 (+ 317us) client_negotiation.cc:769] Sending connection context
0812 02:31:43.574275 (+ 227us) client_negotiation.cc:240] Negotiation successful
0812 02:31:43.574671 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":130,"threads_started":1}
I20260812 02:31:43.577498 15589 heartbeater.cc:347] Connected to a master server at 127.14.103.62:46683
I20260812 02:31:43.577895 15589 heartbeater.cc:464] Registering TS with master...
I20260812 02:31:43.578527 15589 heartbeater.cc:511] Master 127.14.103.62:46683 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.103.2@KRBTEST.COM" created.
I20260812 02:31:43.580782 15367 ts_manager.cc:194] Registered new tserver with Master: 9feda69f42f44dc98b594f71d2752286 (127.14.103.1:35585)
I20260812 02:31:43.582579 15367 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.103.1@KRBTEST.COM'} at 127.14.103.1:60603
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.103.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.2.keytab.
Entry for principal kudu/127.14.103.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.2.keytab.
I20260812 02:31:43.599962 14748 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbep4M4/build/debug/bin/kudu
/tmp/dist-test-taskbep4M4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786501898713699-14748-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786501898713699-14748-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786501898713699-14748-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786501898713699-14748-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.103.2:0
--local_ip_for_outbound_sockets=127.14.103.2
--webserver_interface=127.14.103.2
--webserver_port=0
--tserver_master_addrs=127.14.103.62:46683
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.2.keytab
--principal=kudu/127.14.103.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.103.20:40723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786501898713699-14748-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786501898713699-14748-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786501898713699-14748-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:31:43.728961 15596 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:31:43.729319 15596 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:31:43.729404 15596 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:31:43.733734 15596 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:31:43.733880 15596 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:31:43.733989 15596 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.103.2
I20260812 02:31:43.739897 15596 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.103.20:40723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786501898713699-14748-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786501898713699-14748-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.2.keytab
--principal=kudu/127.14.103.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.103.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786501898713699-14748-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.103.2
--webserver_port=0
--tserver_master_addrs=127.14.103.62:46683
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15596
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.103.2
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786501898713699-14748-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:31:43.741122 15596 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:31:43.742194 15596 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:31:43.752785 15601 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:31:43.752823 15604 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:31:43.753054 15596 server_base.cc:1034] running on GCE node
W20260812 02:31:43.752815 15602 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:31:43.753706 15596 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:31:43.754438 15596 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:31:43.755661 15596 hybrid_clock.cc:648] HybridClock initialized: now 1786501903755623 us; error 45 us; skew 500 ppm
Aug 12 02:31:43 dist-test-slave-2w3w krb5kdc[15328](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501903, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:31:43.759246 15596 init.cc:377] Logged in from keytab as kudu/127.14.103.2@KRBTEST.COM (short username kudu)
I20260812 02:31:43.760262 15596 webserver.cc:460] Webserver started at http://127.14.103.2:43249/ using document root <none> and password file <none>
I20260812 02:31:43.760875 15596 fs_manager.cc:359] Metadata directory not provided
I20260812 02:31:43.760969 15596 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:31:43.761209 15596 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:31:43.763758 15596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786501898713699-14748-0/minicluster-data/ts-1/data/instance:
uuid: "e2f44f407e1a439cb3a05e1a6c3048e4"
format_stamp: "Formatted at 2026-08-12 02:31:43 on dist-test-slave-2w3w"
server_key: "ea1738742b7f9a8d954f634854609f10"
server_key_iv: "abdcb034bc9702bb74ce0f513ce4dd99"
server_key_version: "encryptionkey@0"
I20260812 02:31:43.764684 15596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786501898713699-14748-0/minicluster-data/ts-1/wal/instance:
uuid: "e2f44f407e1a439cb3a05e1a6c3048e4"
format_stamp: "Formatted at 2026-08-12 02:31:43 on dist-test-slave-2w3w"
server_key: "ea1738742b7f9a8d954f634854609f10"
server_key_iv: "abdcb034bc9702bb74ce0f513ce4dd99"
server_key_version: "encryptionkey@0"
I20260812 02:31:43.770746 15596 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260812 02:31:43.774075 15611 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:31:43.776623 15596 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 02:31:43.777448 15596 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786501898713699-14748-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786501898713699-14748-0/minicluster-data/ts-1/wal
uuid: "e2f44f407e1a439cb3a05e1a6c3048e4"
format_stamp: "Formatted at 2026-08-12 02:31:43 on dist-test-slave-2w3w"
server_key: "ea1738742b7f9a8d954f634854609f10"
server_key_iv: "abdcb034bc9702bb74ce0f513ce4dd99"
server_key_version: "encryptionkey@0"
I20260812 02:31:43.777664 15596 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786501898713699-14748-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786501898713699-14748-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786501898713699-14748-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:31:43.797505 15596 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:31:43.801589 15596 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:31:43.801860 15596 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:31:43.802592 15596 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:31:43.803738 15596 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:31:43.803831 15596 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:31:43.803912 15596 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:31:43.803960 15596 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:31:43.816638 15596 rpc_server.cc:307] RPC server started. Bound to: 127.14.103.2:34905
I20260812 02:31:43.816660 15731 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.103.2:34905 every 8 connection(s)
I20260812 02:31:43.817701 15596 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786501898713699-14748-0/minicluster-data/ts-1/data/info.pb
I20260812 02:31:43.819347 14748 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbep4M4/build/debug/bin/kudu as pid 15596
I20260812 02:31:43.819461 14748 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786501898713699-14748-0/minicluster-data/ts-1/wal/instance
I20260812 02:31:43.819741 14748 external_mini_cluster.cc:1428] Setting key c03d125e0155b0a7bf6549627e4ab53a
Aug 12 02:31:43 dist-test-slave-2w3w krb5kdc[15328](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501903, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.2@KRBTEST.COM for kudu/127.14.103.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.14.103.3@KRBTEST.COM; defaulting to no policy
I20260812 02:31:43.836148 15420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:31:43.820617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.2:60815 (local address 127.14.103.62:46683)
0812 02:31:43.820775 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0812 02:31:43.820780 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:31:43.821996 (+ 1216us) server_negotiation.cc:407] Connection header received
0812 02:31:43.824082 (+ 2086us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:31:43.824088 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:31:43.824168 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:31:43.824276 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:31:43.825842 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:31:43.826552 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:31:43.827952 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:31:43.828209 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:31:43.828329 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:31:43.831812 (+ 3483us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:31:43.831818 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:31:43.831821 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:31:43.831856 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:31:43.833689 (+ 1833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:31:43.834353 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:31:43.834356 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:31:43.834358 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:31:43.834449 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:31:43.834850 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:31:43.834854 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:31:43.834856 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:31:43.835138 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:31:43.835251 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0812 02:31:43.835823 (+ 572us) server_negotiation.cc:299] Negotiation successful
0812 02:31:43.835976 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.103.3@KRBTEST.COM" created.
I20260812 02:31:43.837237 15734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:31:43.821037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:46683 (local address 127.14.103.2:60815)
0812 02:31:43.821635 (+ 598us) negotiation.cc:107] Waiting for socket to connect
0812 02:31:43.821701 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0812 02:31:43.823663 (+ 1962us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:31:43.824475 (+ 812us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:31:43.824483 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:31:43.824891 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:31:43.825648 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:31:43.825665 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:31:43.826711 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:31:43.826719 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:31:43.827794 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:31:43.827806 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:31:43.828395 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:31:43.828400 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:31:43.828584 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:31:43.829335 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:31:43.829355 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:31:43.831568 (+ 2213us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:31:43.833861 (+ 2293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:31:43.833869 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:31:43.833891 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:31:43.834227 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:31:43.834548 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:31:43.834551 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:31:43.834554 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:31:43.834723 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:31:43.835251 (+ 528us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:31:43.835260 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:31:43.835553 (+ 293us) client_negotiation.cc:769] Sending connection context
0812 02:31:43.835755 (+ 202us) client_negotiation.cc:240] Negotiation successful
0812 02:31:43.836145 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":189,"threads_started":1}
I20260812 02:31:43.838809 15732 heartbeater.cc:347] Connected to a master server at 127.14.103.62:46683
I20260812 02:31:43.839272 15732 heartbeater.cc:464] Registering TS with master...
I20260812 02:31:43.839886 15732 heartbeater.cc:511] Master 127.14.103.62:46683 requested a full tablet report, sending...
I20260812 02:31:43.841480 15367 ts_manager.cc:194] Registered new tserver with Master: e2f44f407e1a439cb3a05e1a6c3048e4 (127.14.103.2:34905)
I20260812 02:31:43.842519 15367 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.103.2@KRBTEST.COM'} at 127.14.103.2:60815
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.103.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.3.keytab.
Entry for principal kudu/127.14.103.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.3.keytab.
I20260812 02:31:43.860705 14748 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbep4M4/build/debug/bin/kudu
/tmp/dist-test-taskbep4M4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786501898713699-14748-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786501898713699-14748-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786501898713699-14748-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786501898713699-14748-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.103.3:0
--local_ip_for_outbound_sockets=127.14.103.3
--webserver_interface=127.14.103.3
--webserver_port=0
--tserver_master_addrs=127.14.103.62:46683
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.3.keytab
--principal=kudu/127.14.103.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.103.20:40723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786501898713699-14748-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786501898713699-14748-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786501898713699-14748-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:31:44.019032 15740 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:31:44.020009 15740 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:31:44.020423 15740 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:31:44.027784 15740 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:31:44.027894 15740 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:31:44.027966 15740 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.103.3
I20260812 02:31:44.035236 15740 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.103.20:40723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786501898713699-14748-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786501898713699-14748-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.3.keytab
--principal=kudu/127.14.103.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.103.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786501898713699-14748-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.103.3
--webserver_port=0
--tserver_master_addrs=127.14.103.62:46683
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15740
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.103.3
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786501898713699-14748-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:31:44.036913 15740 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:31:44.038146 15740 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:31:44.050168 15745 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:31:44.050168 15748 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:31:44.050318 15740 server_base.cc:1034] running on GCE node
W20260812 02:31:44.050573 15746 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:31:44.051149 15740 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:31:44.051998 15740 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:31:44.053256 15740 hybrid_clock.cc:648] HybridClock initialized: now 1786501904053214 us; error 62 us; skew 500 ppm
Aug 12 02:31:44 dist-test-slave-2w3w krb5kdc[15328](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501904, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:31:44.057060 15740 init.cc:377] Logged in from keytab as kudu/127.14.103.3@KRBTEST.COM (short username kudu)
I20260812 02:31:44.058514 15740 webserver.cc:460] Webserver started at http://127.14.103.3:43561/ using document root <none> and password file <none>
I20260812 02:31:44.059288 15740 fs_manager.cc:359] Metadata directory not provided
I20260812 02:31:44.059394 15740 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:31:44.059654 15740 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:31:44.062392 15740 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786501898713699-14748-0/minicluster-data/ts-2/data/instance:
uuid: "6cf8d78a747047dbb6b6601d728ef280"
format_stamp: "Formatted at 2026-08-12 02:31:44 on dist-test-slave-2w3w"
server_key: "f07fc084c44dc169b01566af95044ff1"
server_key_iv: "7a0601a07a5323a4b0a81b31a453e90b"
server_key_version: "encryptionkey@0"
I20260812 02:31:44.064802 15740 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786501898713699-14748-0/minicluster-data/ts-2/wal/instance:
uuid: "6cf8d78a747047dbb6b6601d728ef280"
format_stamp: "Formatted at 2026-08-12 02:31:44 on dist-test-slave-2w3w"
server_key: "f07fc084c44dc169b01566af95044ff1"
server_key_iv: "7a0601a07a5323a4b0a81b31a453e90b"
server_key_version: "encryptionkey@0"
I20260812 02:31:44.070602 15740 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260812 02:31:44.074208 15756 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:31:44.076447 15740 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 02:31:44.076649 15740 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786501898713699-14748-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786501898713699-14748-0/minicluster-data/ts-2/wal
uuid: "6cf8d78a747047dbb6b6601d728ef280"
format_stamp: "Formatted at 2026-08-12 02:31:44 on dist-test-slave-2w3w"
server_key: "f07fc084c44dc169b01566af95044ff1"
server_key_iv: "7a0601a07a5323a4b0a81b31a453e90b"
server_key_version: "encryptionkey@0"
I20260812 02:31:44.077023 15740 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786501898713699-14748-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786501898713699-14748-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786501898713699-14748-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:31:44.100135 15740 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:31:44.104269 15740 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:31:44.104521 15740 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:31:44.105486 15740 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:31:44.106562 15740 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:31:44.106650 15740 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:31:44.106737 15740 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:31:44.106783 15740 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:31:44.119541 15740 rpc_server.cc:307] RPC server started. Bound to: 127.14.103.3:38103
I20260812 02:31:44.119588 15881 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.103.3:38103 every 8 connection(s)
I20260812 02:31:44.120570 15740 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786501898713699-14748-0/minicluster-data/ts-2/data/info.pb
I20260812 02:31:44.121479 14748 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbep4M4/build/debug/bin/kudu as pid 15740
I20260812 02:31:44.121630 14748 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786501898713699-14748-0/minicluster-data/ts-2/wal/instance
I20260812 02:31:44.121929 14748 external_mini_cluster.cc:1428] Setting key da55eaaeee67eb439a3f4c85bf2e65db
Aug 12 02:31:44 dist-test-slave-2w3w krb5kdc[15328](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501904, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.3@KRBTEST.COM for kudu/127.14.103.62@KRBTEST.COM
I20260812 02:31:44.137741 15420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:31:44.122823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.3:40627 (local address 127.14.103.62:46683)
0812 02:31:44.123292 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0812 02:31:44.123297 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:31:44.124117 (+ 820us) server_negotiation.cc:407] Connection header received
0812 02:31:44.125711 (+ 1594us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:31:44.125714 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:31:44.125775 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:31:44.125854 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:31:44.127219 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:31:44.127840 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:31:44.129046 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:31:44.129308 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:31:44.129474 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:31:44.133087 (+ 3613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:31:44.133091 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:31:44.133093 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:31:44.133136 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:31:44.134905 (+ 1769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:31:44.135736 (+ 831us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:31:44.135740 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:31:44.135742 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:31:44.135866 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:31:44.136399 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:31:44.136402 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:31:44.136403 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:31:44.136699 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:31:44.136794 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0812 02:31:44.137481 (+ 687us) server_negotiation.cc:299] Negotiation successful
0812 02:31:44.137612 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":383}
I20260812 02:31:44.138686 15884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:31:44.123408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:46683 (local address 127.14.103.3:40627)
0812 02:31:44.123949 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0812 02:31:44.124010 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0812 02:31:44.125442 (+ 1432us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:31:44.126071 (+ 629us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:31:44.126078 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:31:44.126495 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:31:44.127039 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:31:44.127054 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:31:44.127982 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:31:44.127984 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:31:44.128912 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:31:44.128922 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:31:44.129535 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:31:44.129538 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:31:44.129697 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:31:44.130664 (+ 967us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:31:44.130682 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:31:44.132823 (+ 2141us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:31:44.135142 (+ 2319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:31:44.135151 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:31:44.135173 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:31:44.135627 (+ 454us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:31:44.136006 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:31:44.136010 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:31:44.136012 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:31:44.136231 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:31:44.136851 (+ 620us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:31:44.136859 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:31:44.137217 (+ 358us) client_negotiation.cc:769] Sending connection context
0812 02:31:44.137402 (+ 185us) client_negotiation.cc:240] Negotiation successful
0812 02:31:44.137722 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":168,"threads_started":1}
I20260812 02:31:44.140136 15882 heartbeater.cc:347] Connected to a master server at 127.14.103.62:46683
I20260812 02:31:44.140385 15882 heartbeater.cc:464] Registering TS with master...
I20260812 02:31:44.140955 15882 heartbeater.cc:511] Master 127.14.103.62:46683 requested a full tablet report, sending...
I20260812 02:31:44.142361 15367 ts_manager.cc:194] Registered new tserver with Master: 6cf8d78a747047dbb6b6601d728ef280 (127.14.103.3:38103)
I20260812 02:31:44.143122 15367 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.103.3@KRBTEST.COM'} at 127.14.103.3:40627
I20260812 02:31:44.149212 14748 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 02:31:44.164737 15420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:31:44.153872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49158 (local address 127.14.103.62:46683)
0812 02:31:44.154083 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0812 02:31:44.154087 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:31:44.154223 (+ 136us) server_negotiation.cc:407] Connection header received
0812 02:31:44.154314 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:31:44.154317 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:31:44.154368 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:31:44.154447 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:31:44.155282 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:31:44.155858 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:31:44.156836 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:31:44.157063 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:31:44.157240 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:31:44.158789 (+ 1549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:31:44.158793 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:31:44.158795 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:31:44.158825 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:31:44.161261 (+ 2436us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:31:44.162127 (+ 866us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:31:44.162131 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:31:44.162132 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:31:44.162242 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:31:44.162632 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:31:44.162635 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:31:44.162637 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:31:44.163623 (+ 986us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:31:44.163779 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0812 02:31:44.164344 (+ 565us) server_negotiation.cc:299] Negotiation successful
0812 02:31:44.164555 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260812 02:31:44.171777 15367 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49158:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 02:31:44.175980 15367 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 02:31:44.196689 15897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:31:44.190114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46736 (local address 127.14.103.1:35585)
0812 02:31:44.191101 (+ 987us) server_negotiation.cc:206] Beginning negotiation
0812 02:31:44.191110 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 02:31:44.191136 (+ 26us) server_negotiation.cc:407] Connection header received
0812 02:31:44.191209 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:31:44.191213 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:31:44.191432 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:31:44.191597 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:31:44.192186 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:31:44.193316 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:31:44.194767 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:31:44.195678 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:31:44.195813 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:31:44.196092 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0812 02:31:44.196398 (+ 306us) server_negotiation.cc:299] Negotiation successful
0812 02:31:44.196518 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":882,"thread_start_us":135,"threads_started":1}
I20260812 02:31:44.196707 15898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:31:44.189633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.1:35585 (local address 127.0.0.1:46736)
0812 02:31:44.190669 (+ 1036us) negotiation.cc:107] Waiting for socket to connect
0812 02:31:44.190683 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 02:31:44.190789 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:31:44.191774 (+ 985us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:31:44.191776 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:31:44.191792 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:31:44.192033 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:31:44.192042 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:31:44.193435 (+ 1393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:31:44.193441 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:31:44.194617 (+ 1176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:31:44.194627 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:31:44.195873 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:31:44.195876 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:31:44.196031 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:31:44.196082 (+ 51us) client_negotiation.cc:769] Sending connection context
0812 02:31:44.196355 (+ 273us) client_negotiation.cc:240] Negotiation successful
0812 02:31:44.196533 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":970,"mutex_wait_us":39,"thread_start_us":111,"threads_started":1}
I20260812 02:31:44.197324 15896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:31:44.189581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48142 (local address 127.14.103.2:34905)
0812 02:31:44.190326 (+ 745us) server_negotiation.cc:206] Beginning negotiation
0812 02:31:44.190339 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0812 02:31:44.190364 (+ 25us) server_negotiation.cc:407] Connection header received
0812 02:31:44.191196 (+ 832us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:31:44.191202 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:31:44.191423 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:31:44.191596 (+ 173us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:31:44.191934 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:31:44.193183 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:31:44.195060 (+ 1877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:31:44.195875 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:31:44.196032 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:31:44.196154 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0812 02:31:44.196527 (+ 373us) server_negotiation.cc:299] Negotiation successful
0812 02:31:44.196639 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":632,"thread_start_us":168,"threads_started":1}
I20260812 02:31:44.197767 15895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:31:44.189396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.2:34905 (local address 127.0.0.1:48142)
0812 02:31:44.190203 (+ 807us) negotiation.cc:107] Waiting for socket to connect
0812 02:31:44.190264 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0812 02:31:44.190411 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:31:44.191604 (+ 1193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:31:44.191610 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:31:44.191636 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:31:44.191787 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:31:44.191796 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:31:44.193321 (+ 1525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:31:44.193324 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:31:44.194892 (+ 1568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:31:44.194904 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:31:44.196053 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:31:44.196056 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:31:44.196260 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:31:44.196305 (+ 45us) client_negotiation.cc:769] Sending connection context
0812 02:31:44.196432 (+ 127us) client_negotiation.cc:240] Negotiation successful
0812 02:31:44.197626 (+ 1194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":569,"mutex_wait_us":1309,"thread_start_us":118,"threads_started":1}
I20260812 02:31:44.197921 15894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:31:44.190326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.3:38103 (local address 127.0.0.1:54614)
0812 02:31:44.190719 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0812 02:31:44.190732 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 02:31:44.190809 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:31:44.191935 (+ 1126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:31:44.191938 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:31:44.191954 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:31:44.192125 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:31:44.192136 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:31:44.193190 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:31:44.193196 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:31:44.194895 (+ 1699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:31:44.194904 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:31:44.197508 (+ 2604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:31:44.197512 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:31:44.197608 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:31:44.197642 (+ 34us) client_negotiation.cc:769] Sending connection context
0812 02:31:44.197691 (+ 49us) client_negotiation.cc:240] Negotiation successful
0812 02:31:44.197773 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"mutex_wait_us":98,"thread_start_us":108,"threads_started":1}
I20260812 02:31:44.198598 15899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:31:44.190669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54614 (local address 127.14.103.3:38103)
0812 02:31:44.191487 (+ 818us) server_negotiation.cc:206] Beginning negotiation
0812 02:31:44.191497 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 02:31:44.191516 (+ 19us) server_negotiation.cc:407] Connection header received
0812 02:31:44.191604 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:31:44.191608 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:31:44.191779 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:31:44.192086 (+ 307us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:31:44.192250 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:31:44.193048 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:31:44.196199 (+ 3151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:31:44.197353 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:31:44.198063 (+ 710us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:31:44.198260 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0812 02:31:44.198318 (+ 58us) server_negotiation.cc:299] Negotiation successful
0812 02:31:44.198429 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":698,"thread_start_us":146,"threads_started":1}
I20260812 02:31:44.200788 15514 tablet_service.cc:1467] Processing CreateTablet for tablet c6576ce34ef94f6c808c33c3c212a66b (DEFAULT_TABLE table=test-table [id=2bc314ddbf854bd7873f5545d2c3b7e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:31:44.201062 15807 tablet_service.cc:1467] Processing CreateTablet for tablet c6576ce34ef94f6c808c33c3c212a66b (DEFAULT_TABLE table=test-table [id=2bc314ddbf854bd7873f5545d2c3b7e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:31:44.201224 15662 tablet_service.cc:1467] Processing CreateTablet for tablet c6576ce34ef94f6c808c33c3c212a66b (DEFAULT_TABLE table=test-table [id=2bc314ddbf854bd7873f5545d2c3b7e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:31:44.202277 15662 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6576ce34ef94f6c808c33c3c212a66b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:31:44.202278 15514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6576ce34ef94f6c808c33c3c212a66b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:31:44.202313 15807 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6576ce34ef94f6c808c33c3c212a66b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:31:44.209408 15900 tablet_bootstrap.cc:492] T c6576ce34ef94f6c808c33c3c212a66b P 6cf8d78a747047dbb6b6601d728ef280: Bootstrap starting.
I20260812 02:31:44.210143 15902 tablet_bootstrap.cc:492] T c6576ce34ef94f6c808c33c3c212a66b P e2f44f407e1a439cb3a05e1a6c3048e4: Bootstrap starting.
I20260812 02:31:44.211366 15901 tablet_bootstrap.cc:492] T c6576ce34ef94f6c808c33c3c212a66b P 9feda69f42f44dc98b594f71d2752286: Bootstrap starting.
I20260812 02:31:44.213058 15900 tablet_bootstrap.cc:654] T c6576ce34ef94f6c808c33c3c212a66b P 6cf8d78a747047dbb6b6601d728ef280: Neither blocks nor log segments found. Creating new log.
I20260812 02:31:44.213436 15902 tablet_bootstrap.cc:654] T c6576ce34ef94f6c808c33c3c212a66b P e2f44f407e1a439cb3a05e1a6c3048e4: Neither blocks nor log segments found. Creating new log.
I20260812 02:31:44.213927 15901 tablet_bootstrap.cc:654] T c6576ce34ef94f6c808c33c3c212a66b P 9feda69f42f44dc98b594f71d2752286: Neither blocks nor log segments found. Creating new log.
I20260812 02:31:44.214684 15900 log.cc:824] T c6576ce34ef94f6c808c33c3c212a66b P 6cf8d78a747047dbb6b6601d728ef280: Log is configured to *not* fsync() on all Append() calls
I20260812 02:31:44.214782 15902 log.cc:824] T c6576ce34ef94f6c808c33c3c212a66b P e2f44f407e1a439cb3a05e1a6c3048e4: Log is configured to *not* fsync() on all Append() calls
I20260812 02:31:44.214782 15901 log.cc:824] T c6576ce34ef94f6c808c33c3c212a66b P 9feda69f42f44dc98b594f71d2752286: Log is configured to *not* fsync() on all Append() calls
I20260812 02:31:44.217825 15900 tablet_bootstrap.cc:492] T c6576ce34ef94f6c808c33c3c212a66b P 6cf8d78a747047dbb6b6601d728ef280: No bootstrap required, opened a new log
I20260812 02:31:44.217935 15901 tablet_bootstrap.cc:492] T c6576ce34ef94f6c808c33c3c212a66b P 9feda69f42f44dc98b594f71d2752286: No bootstrap required, opened a new log
I20260812 02:31:44.218158 15900 ts_tablet_manager.cc:1397] T c6576ce34ef94f6c808c33c3c212a66b P 6cf8d78a747047dbb6b6601d728ef280: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260812 02:31:44.219156 15901 ts_tablet_manager.cc:1397] T c6576ce34ef94f6c808c33c3c212a66b P 9feda69f42f44dc98b594f71d2752286: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.002s
I20260812 02:31:44.219798 15902 tablet_bootstrap.cc:492] T c6576ce34ef94f6c808c33c3c212a66b P e2f44f407e1a439cb3a05e1a6c3048e4: No bootstrap required, opened a new log
I20260812 02:31:44.220008 15902 ts_tablet_manager.cc:1397] T c6576ce34ef94f6c808c33c3c212a66b P e2f44f407e1a439cb3a05e1a6c3048e4: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.002s
I20260812 02:31:44.222486 15900 raft_consensus.cc:348] T c6576ce34ef94f6c808c33c3c212a66b P 6cf8d78a747047dbb6b6601d728ef280 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2f44f407e1a439cb3a05e1a6c3048e4" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 34905 } } peers { permanent_uuid: "9feda69f42f44dc98b594f71d2752286" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 35585 } } peers { permanent_uuid: "6cf8d78a747047dbb6b6601d728ef280" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 38103 } }
I20260812 02:31:44.222801 15900 raft_consensus.cc:374] T c6576ce34ef94f6c808c33c3c212a66b P 6cf8d78a747047dbb6b6601d728ef280 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:31:44.222844 15900 raft_consensus.cc:729] T c6576ce34ef94f6c808c33c3c212a66b P 6cf8d78a747047dbb6b6601d728ef280 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cf8d78a747047dbb6b6601d728ef280, State: Initialized, Role: FOLLOWER
I20260812 02:31:44.223431 15900 consensus_queue.cc:260] T c6576ce34ef94f6c808c33c3c212a66b P 6cf8d78a747047dbb6b6601d728ef280 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2f44f407e1a439cb3a05e1a6c3048e4" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 34905 } } peers { permanent_uuid: "9feda69f42f44dc98b594f71d2752286" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 35585 } } peers { permanent_uuid: "6cf8d78a747047dbb6b6601d728ef280" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 38103 } }
I20260812 02:31:44.224380 15901 raft_consensus.cc:348] T c6576ce34ef94f6c808c33c3c212a66b P 9feda69f42f44dc98b594f71d2752286 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2f44f407e1a439cb3a05e1a6c3048e4" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 34905 } } peers { permanent_uuid: "9feda69f42f44dc98b594f71d2752286" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 35585 } } peers { permanent_uuid: "6cf8d78a747047dbb6b6601d728ef280" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 38103 } }
I20260812 02:31:44.224380 15902 raft_consensus.cc:348] T c6576ce34ef94f6c808c33c3c212a66b P e2f44f407e1a439cb3a05e1a6c3048e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2f44f407e1a439cb3a05e1a6c3048e4" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 34905 } } peers { permanent_uuid: "9feda69f42f44dc98b594f71d2752286" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 35585 } } peers { permanent_uuid: "6cf8d78a747047dbb6b6601d728ef280" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 38103 } }
I20260812 02:31:44.224627 15900 ts_tablet_manager.cc:1428] T c6576ce34ef94f6c808c33c3c212a66b P 6cf8d78a747047dbb6b6601d728ef280: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260812 02:31:44.224670 15901 raft_consensus.cc:374] T c6576ce34ef94f6c808c33c3c212a66b P 9feda69f42f44dc98b594f71d2752286 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:31:44.224670 15902 raft_consensus.cc:374] T c6576ce34ef94f6c808c33c3c212a66b P e2f44f407e1a439cb3a05e1a6c3048e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:31:44.224730 15902 raft_consensus.cc:729] T c6576ce34ef94f6c808c33c3c212a66b P e2f44f407e1a439cb3a05e1a6c3048e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2f44f407e1a439cb3a05e1a6c3048e4, State: Initialized, Role: FOLLOWER
I20260812 02:31:44.224730 15901 raft_consensus.cc:729] T c6576ce34ef94f6c808c33c3c212a66b P 9feda69f42f44dc98b594f71d2752286 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9feda69f42f44dc98b594f71d2752286, State: Initialized, Role: FOLLOWER
I20260812 02:31:44.225117 15882 heartbeater.cc:503] Master 127.14.103.62:46683 was elected leader, sending a full tablet report...
I20260812 02:31:44.225270 15902 consensus_queue.cc:260] T c6576ce34ef94f6c808c33c3c212a66b P e2f44f407e1a439cb3a05e1a6c3048e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2f44f407e1a439cb3a05e1a6c3048e4" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 34905 } } peers { permanent_uuid: "9feda69f42f44dc98b594f71d2752286" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 35585 } } peers { permanent_uuid: "6cf8d78a747047dbb6b6601d728ef280" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 38103 } }
I20260812 02:31:44.225270 15901 consensus_queue.cc:260] T c6576ce34ef94f6c808c33c3c212a66b P 9feda69f42f44dc98b594f71d2752286 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2f44f407e1a439cb3a05e1a6c3048e4" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 34905 } } peers { permanent_uuid: "9feda69f42f44dc98b594f71d2752286" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 35585 } } peers { permanent_uuid: "6cf8d78a747047dbb6b6601d728ef280" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 38103 } }
I20260812 02:31:44.226444 15902 ts_tablet_manager.cc:1428] T c6576ce34ef94f6c808c33c3c212a66b P e2f44f407e1a439cb3a05e1a6c3048e4: Time spent starting tablet: real 0.006s user 0.006s sys 0.001s
I20260812 02:31:44.226444 15901 ts_tablet_manager.cc:1428] T c6576ce34ef94f6c808c33c3c212a66b P 9feda69f42f44dc98b594f71d2752286: Time spent starting tablet: real 0.007s user 0.005s sys 0.002s
I20260812 02:31:44.226820 15589 heartbeater.cc:503] Master 127.14.103.62:46683 was elected leader, sending a full tablet report...
I20260812 02:31:44.226820 15732 heartbeater.cc:503] Master 127.14.103.62:46683 was elected leader, sending a full tablet report...
W20260812 02:31:44.310492 15590 tablet.cc:2369] T c6576ce34ef94f6c808c33c3c212a66b P 9feda69f42f44dc98b594f71d2752286: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 02:31:44.319686 15733 tablet.cc:2369] T c6576ce34ef94f6c808c33c3c212a66b P e2f44f407e1a439cb3a05e1a6c3048e4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 02:31:44.371819 15883 tablet.cc:2369] T c6576ce34ef94f6c808c33c3c212a66b P 6cf8d78a747047dbb6b6601d728ef280: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 02:31:44.378232 15909 raft_consensus.cc:482] T c6576ce34ef94f6c808c33c3c212a66b P e2f44f407e1a439cb3a05e1a6c3048e4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 02:31:44.378538 15909 raft_consensus.cc:504] T c6576ce34ef94f6c808c33c3c212a66b P e2f44f407e1a439cb3a05e1a6c3048e4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2f44f407e1a439cb3a05e1a6c3048e4" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 34905 } } peers { permanent_uuid: "9feda69f42f44dc98b594f71d2752286" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 35585 } } peers { permanent_uuid: "6cf8d78a747047dbb6b6601d728ef280" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 38103 } }
I20260812 02:31:44.379784 15909 leader_election.cc:288] T c6576ce34ef94f6c808c33c3c212a66b P e2f44f407e1a439cb3a05e1a6c3048e4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9feda69f42f44dc98b594f71d2752286 (127.14.103.1:35585), 6cf8d78a747047dbb6b6601d728ef280 (127.14.103.3:38103)
I20260812 02:31:44.384268 15899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:31:44.380246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.2:35601 (local address 127.14.103.3:38103)
0812 02:31:44.380446 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0812 02:31:44.380451 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:31:44.380658 (+ 207us) server_negotiation.cc:407] Connection header received
0812 02:31:44.380818 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:31:44.380821 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:31:44.380875 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:31:44.380962 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:31:44.381392 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:31:44.381966 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:31:44.383261 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:31:44.383682 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:31:44.383813 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:31:44.383869 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0812 02:31:44.384056 (+ 187us) server_negotiation.cc:299] Negotiation successful
0812 02:31:44.384134 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260812 02:31:44.384294 15911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:31:44.380106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.3:38103 (local address 127.14.103.2:35601)
0812 02:31:44.380546 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0812 02:31:44.380558 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 02:31:44.380707 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:31:44.380973 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:31:44.380976 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:31:44.380991 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:31:44.381204 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:31:44.381211 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:31:44.382078 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:31:44.382083 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:31:44.383140 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:31:44.383150 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:31:44.383811 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:31:44.383815 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:31:44.383932 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:31:44.383951 (+ 19us) client_negotiation.cc:769] Sending connection context
0812 02:31:44.384038 (+ 87us) client_negotiation.cc:240] Negotiation successful
0812 02:31:44.384123 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":381,"mutex_wait_us":71,"spinlock_wait_cycles":4864,"thread_start_us":94,"threads_started":1}
I20260812 02:31:44.384465 15897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:31:44.380281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.2:56345 (local address 127.14.103.1:35585)
0812 02:31:44.380463 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0812 02:31:44.380468 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:31:44.380601 (+ 133us) server_negotiation.cc:407] Connection header received
0812 02:31:44.380781 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:31:44.380784 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:31:44.380829 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:31:44.380985 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:31:44.381500 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:31:44.382157 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:31:44.383439 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:31:44.383879 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:31:44.384018 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:31:44.384079 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0812 02:31:44.384258 (+ 179us) server_negotiation.cc:299] Negotiation successful
0812 02:31:44.384341 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260812 02:31:44.384485 15910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:31:44.380095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.1:35585 (local address 127.14.103.2:56345)
0812 02:31:44.380501 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0812 02:31:44.380520 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 02:31:44.380661 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:31:44.380998 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:31:44.381001 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:31:44.381041 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:31:44.381313 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:31:44.381323 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:31:44.382330 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:31:44.382333 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:31:44.383310 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:31:44.383320 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:31:44.384012 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:31:44.384015 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:31:44.384129 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:31:44.384153 (+ 24us) client_negotiation.cc:769] Sending connection context
0812 02:31:44.384239 (+ 86us) client_negotiation.cc:240] Negotiation successful
0812 02:31:44.384321 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":333,"mutex_wait_us":33,"spinlock_wait_cycles":2048,"thread_start_us":80,"threads_started":1}
I20260812 02:31:44.385201 15829 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6576ce34ef94f6c808c33c3c212a66b" candidate_uuid: "e2f44f407e1a439cb3a05e1a6c3048e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cf8d78a747047dbb6b6601d728ef280" is_pre_election: true
I20260812 02:31:44.385281 15541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6576ce34ef94f6c808c33c3c212a66b" candidate_uuid: "e2f44f407e1a439cb3a05e1a6c3048e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9feda69f42f44dc98b594f71d2752286" is_pre_election: true
I20260812 02:31:44.385668 15829 raft_consensus.cc:2454] T c6576ce34ef94f6c808c33c3c212a66b P 6cf8d78a747047dbb6b6601d728ef280 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2f44f407e1a439cb3a05e1a6c3048e4 in term 0.
I20260812 02:31:44.386308 15615 leader_election.cc:302] T c6576ce34ef94f6c808c33c3c212a66b P e2f44f407e1a439cb3a05e1a6c3048e4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6cf8d78a747047dbb6b6601d728ef280, e2f44f407e1a439cb3a05e1a6c3048e4; no voters:
I20260812 02:31:44.386658 15909 raft_consensus.cc:2781] T c6576ce34ef94f6c808c33c3c212a66b P e2f44f407e1a439cb3a05e1a6c3048e4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 02:31:44.386750 15909 raft_consensus.cc:482] T c6576ce34ef94f6c808c33c3c212a66b P e2f44f407e1a439cb3a05e1a6c3048e4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 02:31:44.386783 15909 raft_consensus.cc:3037] T c6576ce34ef94f6c808c33c3c212a66b P e2f44f407e1a439cb3a05e1a6c3048e4 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:31:44.388068 15909 raft_consensus.cc:504] T c6576ce34ef94f6c808c33c3c212a66b P e2f44f407e1a439cb3a05e1a6c3048e4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2f44f407e1a439cb3a05e1a6c3048e4" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 34905 } } peers { permanent_uuid: "9feda69f42f44dc98b594f71d2752286" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 35585 } } peers { permanent_uuid: "6cf8d78a747047dbb6b6601d728ef280" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 38103 } }
I20260812 02:31:44.388229 15541 raft_consensus.cc:2454] T c6576ce34ef94f6c808c33c3c212a66b P 9feda69f42f44dc98b594f71d2752286 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2f44f407e1a439cb3a05e1a6c3048e4 in term 0.
I20260812 02:31:44.388525 15909 leader_election.cc:288] T c6576ce34ef94f6c808c33c3c212a66b P e2f44f407e1a439cb3a05e1a6c3048e4 [CANDIDATE]: Term 1 election: Requested vote from peers 9feda69f42f44dc98b594f71d2752286 (127.14.103.1:35585), 6cf8d78a747047dbb6b6601d728ef280 (127.14.103.3:38103)
I20260812 02:31:44.388940 15541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6576ce34ef94f6c808c33c3c212a66b" candidate_uuid: "e2f44f407e1a439cb3a05e1a6c3048e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9feda69f42f44dc98b594f71d2752286"
I20260812 02:31:44.389112 15541 raft_consensus.cc:3037] T c6576ce34ef94f6c808c33c3c212a66b P 9feda69f42f44dc98b594f71d2752286 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:31:44.389253 15829 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6576ce34ef94f6c808c33c3c212a66b" candidate_uuid: "e2f44f407e1a439cb3a05e1a6c3048e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cf8d78a747047dbb6b6601d728ef280"
I20260812 02:31:44.389424 15829 raft_consensus.cc:3037] T c6576ce34ef94f6c808c33c3c212a66b P 6cf8d78a747047dbb6b6601d728ef280 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:31:44.390362 15541 raft_consensus.cc:2454] T c6576ce34ef94f6c808c33c3c212a66b P 9feda69f42f44dc98b594f71d2752286 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2f44f407e1a439cb3a05e1a6c3048e4 in term 1.
I20260812 02:31:44.390547 15829 raft_consensus.cc:2454] T c6576ce34ef94f6c808c33c3c212a66b P 6cf8d78a747047dbb6b6601d728ef280 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2f44f407e1a439cb3a05e1a6c3048e4 in term 1.
I20260812 02:31:44.390821 15612 leader_election.cc:302] T c6576ce34ef94f6c808c33c3c212a66b P e2f44f407e1a439cb3a05e1a6c3048e4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9feda69f42f44dc98b594f71d2752286, e2f44f407e1a439cb3a05e1a6c3048e4; no voters:
I20260812 02:31:44.391103 15909 raft_consensus.cc:2781] T c6576ce34ef94f6c808c33c3c212a66b P e2f44f407e1a439cb3a05e1a6c3048e4 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:31:44.391371 15909 raft_consensus.cc:686] T c6576ce34ef94f6c808c33c3c212a66b P e2f44f407e1a439cb3a05e1a6c3048e4 [term 1 LEADER]: Becoming Leader. State: Replica: e2f44f407e1a439cb3a05e1a6c3048e4, State: Running, Role: LEADER
I20260812 02:31:44.391783 15909 consensus_queue.cc:237] T c6576ce34ef94f6c808c33c3c212a66b P e2f44f407e1a439cb3a05e1a6c3048e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2f44f407e1a439cb3a05e1a6c3048e4" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 34905 } } peers { permanent_uuid: "9feda69f42f44dc98b594f71d2752286" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 35585 } } peers { permanent_uuid: "6cf8d78a747047dbb6b6601d728ef280" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 38103 } }
I20260812 02:31:44.395651 15378 catalog_manager.cc:5495] T c6576ce34ef94f6c808c33c3c212a66b P e2f44f407e1a439cb3a05e1a6c3048e4 reported cstate change: term changed from 0 to 1, leader changed from <none> to e2f44f407e1a439cb3a05e1a6c3048e4 (127.14.103.2). New cstate: current_term: 1 leader_uuid: "e2f44f407e1a439cb3a05e1a6c3048e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2f44f407e1a439cb3a05e1a6c3048e4" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 34905 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9feda69f42f44dc98b594f71d2752286" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 35585 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6cf8d78a747047dbb6b6601d728ef280" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 38103 } health_report { overall_health: UNKNOWN } } }
Aug 12 02:31:44 dist-test-slave-2w3w krb5kdc[15328](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501902, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.103.1@KRBTEST.COM
I20260812 02:31:44.447816 15897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:31:44.436220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46748 (local address 127.14.103.1:35585)
0812 02:31:44.436452 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0812 02:31:44.436457 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:31:44.436862 (+ 405us) server_negotiation.cc:407] Connection header received
0812 02:31:44.437052 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:31:44.437056 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:31:44.437130 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:31:44.437224 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:31:44.437990 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:31:44.438530 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:31:44.439496 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:31:44.439724 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:31:44.439840 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:31:44.442927 (+ 3087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:31:44.442951 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:31:44.443012 (+ 61us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:31:44.443052 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:31:44.445388 (+ 2336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:31:44.446066 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:31:44.446072 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:31:44.446074 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:31:44.446211 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:31:44.446588 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:31:44.446595 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:31:44.446599 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:31:44.447018 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:31:44.447254 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0812 02:31:44.447488 (+ 234us) server_negotiation.cc:299] Negotiation successful
0812 02:31:44.447636 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
Aug 12 02:31:44 dist-test-slave-2w3w krb5kdc[15328](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501904, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 12 02:31:44 dist-test-slave-2w3w krb5kdc[15328](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501904, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.14.103.1@KRBTEST.COM
I20260812 02:31:44.483428 15897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:31:44.471991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46756 (local address 127.14.103.1:35585)
0812 02:31:44.472160 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0812 02:31:44.472163 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 02:31:44.472181 (+ 18us) server_negotiation.cc:407] Connection header received
0812 02:31:44.472312 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:31:44.472315 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:31:44.472361 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:31:44.472435 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:31:44.473262 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:31:44.473788 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:31:44.474946 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:31:44.475293 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:31:44.475408 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:31:44.478949 (+ 3541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:31:44.478953 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:31:44.478955 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:31:44.479027 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:31:44.480814 (+ 1787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:31:44.481597 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:31:44.481601 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:31:44.481602 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:31:44.481713 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:31:44.482122 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:31:44.482125 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:31:44.482126 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:31:44.482347 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:31:44.482479 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0812 02:31:44.482858 (+ 379us) server_negotiation.cc:299] Negotiation successful
0812 02:31:44.483286 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
Aug 12 02:31:44 dist-test-slave-2w3w krb5kdc[15328](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501904, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 12 02:31:44 dist-test-slave-2w3w krb5kdc[15328](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501904, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.103.1@KRBTEST.COM
I20260812 02:31:44.514420 15897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:31:44.504747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46758 (local address 127.14.103.1:35585)
0812 02:31:44.504939 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0812 02:31:44.504944 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:31:44.505094 (+ 150us) server_negotiation.cc:407] Connection header received
0812 02:31:44.505283 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:31:44.505286 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:31:44.505333 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:31:44.505420 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:31:44.506183 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:31:44.506713 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:31:44.507658 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:31:44.507881 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:31:44.507991 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:31:44.510768 (+ 2777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:31:44.510772 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:31:44.510774 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:31:44.510807 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:31:44.512589 (+ 1782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:31:44.513127 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:31:44.513130 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:31:44.513131 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:31:44.513218 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:31:44.513542 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:31:44.513544 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:31:44.513545 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:31:44.513723 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:31:44.513833 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0812 02:31:44.514106 (+ 273us) server_negotiation.cc:299] Negotiation successful
0812 02:31:44.514221 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
W20260812 02:31:44.516260 15494 tablet_service.cc:2908] Rejecting scan request for tablet c6576ce34ef94f6c808c33c3c212a66b: Uninitialized: clean time has not yet been initialized
I20260812 02:31:44.519086 14748 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbep4M4/build/debug/bin/kudu with pid 15446
I20260812 02:31:44.527504 14748 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbep4M4/build/debug/bin/kudu with pid 15596
I20260812 02:31:44.535902 14748 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbep4M4/build/debug/bin/kudu with pid 15740
I20260812 02:31:44.544693 14748 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbep4M4/build/debug/bin/kudu with pid 15343
2026-08-12T02:31:44Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2965 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501898713699-14748-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 02:31:44 dist-test-slave-2w3w krb5kdc[15928](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 02:31:44 dist-test-slave-2w3w krb5kdc[15928](info): set up 2 sockets
Aug 12 02:31:44 dist-test-slave-2w3w krb5kdc[15928](info): commencing operation
krb5kdc: starting...
W20260812 02:31:45.700397 14748 mini_kdc.cc:121] Time spent starting KDC: real 1.130s user 0.000s sys 0.009s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501898713699-14748-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501898713699-14748-0/krb5kdc/test-user.keytab.
Aug 12 02:31:45 dist-test-slave-2w3w krb5kdc[15928](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501905, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T02:31:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T02:31:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.103.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.103.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.103.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.62.keytab.
Entry for principal kudu/127.14.103.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.62.keytab.
I20260812 02:31:45.852072 14748 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbep4M4/build/debug/bin/kudu
/tmp/dist-test-taskbep4M4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501898713699-14748-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501898713699-14748-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501898713699-14748-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501898713699-14748-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.103.62:40425
--webserver_interface=127.14.103.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.62.keytab
--principal=kudu/127.14.103.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.103.20:32793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.103.62:40425
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501898713699-14748-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501898713699-14748-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501898713699-14748-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:31:45.975544 15942 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:31:45.975900 15942 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:31:45.975946 15942 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:31:45.980710 15942 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:31:45.980818 15942 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260812 02:31:45.980839 15942 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:31:45.980860 15942 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:31:45.980880 15942 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:31:45.980898 15942 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 02:31:45.987768 15942 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.103.20:32793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501898713699-14748-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501898713699-14748-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.103.62:40425
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.62.keytab
--principal=kudu/127.14.103.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.103.62:40425
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501898713699-14748-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.103.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15942
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501898713699-14748-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:31:45.988933 15942 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:31:45.989986 15942 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:31:45.999380 15951 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:31:45.999577 15948 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:31:45.999629 15947 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:31:46.000012 15942 server_base.cc:1034] running on GCE node
I20260812 02:31:46.000674 15942 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:31:46.001965 15942 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:31:46.003266 15942 hybrid_clock.cc:648] HybridClock initialized: now 1786501906003228 us; error 49 us; skew 500 ppm
Aug 12 02:31:45 dist-test-slave-2w3w krb5kdc[15928](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501905, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:31:46.007236 15942 init.cc:377] Logged in from keytab as kudu/127.14.103.62@KRBTEST.COM (short username kudu)
I20260812 02:31:46.008535 15942 webserver.cc:460] Webserver started at http://127.14.103.62:42683/ using document root <none> and password file <none>
I20260812 02:31:46.009204 15942 fs_manager.cc:359] Metadata directory not provided
I20260812 02:31:46.009289 15942 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:31:46.009539 15942 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:31:46.012423 15942 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501898713699-14748-0/minicluster-data/master-0/data/instance:
uuid: "be19d8ff7b3e4e17a7341b403e1245df"
format_stamp: "Formatted at 2026-08-12 02:31:46 on dist-test-slave-2w3w"
server_key: "b8addf69210715f6491d3f0296b2ab03"
server_key_iv: "d69fe48e12b4df9de9f83c463a1bba80"
server_key_version: "encryptionkey@0"
I20260812 02:31:46.013127 15942 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501898713699-14748-0/minicluster-data/master-0/wal/instance:
uuid: "be19d8ff7b3e4e17a7341b403e1245df"
format_stamp: "Formatted at 2026-08-12 02:31:46 on dist-test-slave-2w3w"
server_key: "b8addf69210715f6491d3f0296b2ab03"
server_key_iv: "d69fe48e12b4df9de9f83c463a1bba80"
server_key_version: "encryptionkey@0"
I20260812 02:31:46.017975 15942 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260812 02:31:46.022498 15958 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:31:46.024313 15942 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 02:31:46.024598 15942 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501898713699-14748-0/minicluster-data/master-0/data,/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501898713699-14748-0/minicluster-data/master-0/wal
uuid: "be19d8ff7b3e4e17a7341b403e1245df"
format_stamp: "Formatted at 2026-08-12 02:31:46 on dist-test-slave-2w3w"
server_key: "b8addf69210715f6491d3f0296b2ab03"
server_key_iv: "d69fe48e12b4df9de9f83c463a1bba80"
server_key_version: "encryptionkey@0"
I20260812 02:31:46.024788 15942 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501898713699-14748-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501898713699-14748-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501898713699-14748-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:31:46.036743 15942 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:31:46.040752 15942 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:31:46.040995 15942 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:31:46.052134 15942 rpc_server.cc:307] RPC server started. Bound to: 127.14.103.62:40425
I20260812 02:31:46.052188 16026 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.103.62:40425 every 8 connection(s)
I20260812 02:31:46.053220 15942 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501898713699-14748-0/minicluster-data/master-0/data/info.pb
I20260812 02:31:46.056915 16027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:31:46.061143 14748 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbep4M4/build/debug/bin/kudu as pid 15942
I20260812 02:31:46.061295 14748 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501898713699-14748-0/minicluster-data/master-0/wal/instance
I20260812 02:31:46.061576 14748 external_mini_cluster.cc:1428] Setting key 9287f5430b2d3fdc63371528bc988129
I20260812 02:31:46.064073 16027 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be19d8ff7b3e4e17a7341b403e1245df: Bootstrap starting.
I20260812 02:31:46.066457 16027 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P be19d8ff7b3e4e17a7341b403e1245df: Neither blocks nor log segments found. Creating new log.
I20260812 02:31:46.067364 16027 log.cc:824] T 00000000000000000000000000000000 P be19d8ff7b3e4e17a7341b403e1245df: Log is configured to *not* fsync() on all Append() calls
I20260812 02:31:46.070132 16027 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be19d8ff7b3e4e17a7341b403e1245df: No bootstrap required, opened a new log
Aug 12 02:31:46 dist-test-slave-2w3w krb5kdc[15928](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501905, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.103.62@KRBTEST.COM
I20260812 02:31:46.075291 16027 raft_consensus.cc:348] T 00000000000000000000000000000000 P be19d8ff7b3e4e17a7341b403e1245df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be19d8ff7b3e4e17a7341b403e1245df" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 40425 } }
I20260812 02:31:46.075627 16027 raft_consensus.cc:374] T 00000000000000000000000000000000 P be19d8ff7b3e4e17a7341b403e1245df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:31:46.075711 16027 raft_consensus.cc:729] T 00000000000000000000000000000000 P be19d8ff7b3e4e17a7341b403e1245df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be19d8ff7b3e4e17a7341b403e1245df, State: Initialized, Role: FOLLOWER
I20260812 02:31:46.076282 16027 consensus_queue.cc:260] T 00000000000000000000000000000000 P be19d8ff7b3e4e17a7341b403e1245df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be19d8ff7b3e4e17a7341b403e1245df" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 40425 } }
I20260812 02:31:46.076510 16027 raft_consensus.cc:388] T 00000000000000000000000000000000 P be19d8ff7b3e4e17a7341b403e1245df [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 02:31:46.076606 16027 raft_consensus.cc:482] T 00000000000000000000000000000000 P be19d8ff7b3e4e17a7341b403e1245df [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 02:31:46.076728 16027 raft_consensus.cc:3037] T 00000000000000000000000000000000 P be19d8ff7b3e4e17a7341b403e1245df [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:31:46.078133 16027 raft_consensus.cc:504] T 00000000000000000000000000000000 P be19d8ff7b3e4e17a7341b403e1245df [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be19d8ff7b3e4e17a7341b403e1245df" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 40425 } }
I20260812 02:31:46.078555 16027 leader_election.cc:302] T 00000000000000000000000000000000 P be19d8ff7b3e4e17a7341b403e1245df [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: be19d8ff7b3e4e17a7341b403e1245df; no voters:
I20260812 02:31:46.079015 16027 leader_election.cc:288] T 00000000000000000000000000000000 P be19d8ff7b3e4e17a7341b403e1245df [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 02:31:46.079238 16032 raft_consensus.cc:2781] T 00000000000000000000000000000000 P be19d8ff7b3e4e17a7341b403e1245df [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:31:46.079504 16030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:31:46.063406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32896 (local address 127.14.103.62:40425)
0812 02:31:46.063966 (+ 560us) server_negotiation.cc:206] Beginning negotiation
0812 02:31:46.063985 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0812 02:31:46.064024 (+ 39us) server_negotiation.cc:407] Connection header received
0812 02:31:46.064857 (+ 833us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:31:46.064875 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:31:46.065239 (+ 364us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:31:46.065598 (+ 359us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:31:46.066471 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:31:46.067688 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:31:46.068966 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:31:46.069527 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:31:46.069684 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:31:46.073377 (+ 3693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:31:46.073384 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:31:46.073401 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:31:46.073438 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:31:46.075500 (+ 2062us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:31:46.076162 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:31:46.076168 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:31:46.076174 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:31:46.076356 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:31:46.076755 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:31:46.076758 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:31:46.076760 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:31:46.077356 (+ 596us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:31:46.077579 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0812 02:31:46.077864 (+ 285us) server_negotiation.cc:299] Negotiation successful
0812 02:31:46.078226 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":162,"threads_started":1}
I20260812 02:31:46.079851 16032 raft_consensus.cc:686] T 00000000000000000000000000000000 P be19d8ff7b3e4e17a7341b403e1245df [term 1 LEADER]: Becoming Leader. State: Replica: be19d8ff7b3e4e17a7341b403e1245df, State: Running, Role: LEADER
I20260812 02:31:46.080482 16027 sys_catalog.cc:564] T 00000000000000000000000000000000 P be19d8ff7b3e4e17a7341b403e1245df [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:31:46.080493 16032 consensus_queue.cc:237] T 00000000000000000000000000000000 P be19d8ff7b3e4e17a7341b403e1245df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be19d8ff7b3e4e17a7341b403e1245df" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 40425 } }
I20260812 02:31:46.081853 16032 sys_catalog.cc:455] T 00000000000000000000000000000000 P be19d8ff7b3e4e17a7341b403e1245df [sys.catalog]: SysCatalogTable state changed. Reason: New leader be19d8ff7b3e4e17a7341b403e1245df. Latest consensus state: current_term: 1 leader_uuid: "be19d8ff7b3e4e17a7341b403e1245df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be19d8ff7b3e4e17a7341b403e1245df" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 40425 } } }
I20260812 02:31:46.081995 16032 sys_catalog.cc:458] T 00000000000000000000000000000000 P be19d8ff7b3e4e17a7341b403e1245df [sys.catalog]: This master's current role is: LEADER
I20260812 02:31:46.082932 16034 sys_catalog.cc:455] T 00000000000000000000000000000000 P be19d8ff7b3e4e17a7341b403e1245df [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "be19d8ff7b3e4e17a7341b403e1245df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be19d8ff7b3e4e17a7341b403e1245df" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 40425 } } }
I20260812 02:31:46.083159 16034 sys_catalog.cc:458] T 00000000000000000000000000000000 P be19d8ff7b3e4e17a7341b403e1245df [sys.catalog]: This master's current role is: LEADER
W20260812 02:31:46.087698 16048 catalog_manager.cc:1558] T 00000000000000000000000000000000 P be19d8ff7b3e4e17a7341b403e1245df: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 02:31:46.087813 16048 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 02:31:46.087980 16041 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 02:31:46.089370 16041 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 02:31:46.098106 16041 catalog_manager.cc:1347] Generated new cluster ID: 451bbffc9315471f9615f22057e90a04
I20260812 02:31:46.098217 16041 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 02:31:46.110427 16041 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 02:31:46.111627 16041 catalog_manager.cc:1504] Loading token signing keys...
I20260812 02:31:46.125303 16041 catalog_manager.cc:5868] T 00000000000000000000000000000000 P be19d8ff7b3e4e17a7341b403e1245df: Generated new TSK 0
I20260812 02:31:46.126295 16041 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.103.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.103.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.103.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.1.keytab.
Entry for principal kudu/127.14.103.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.1.keytab.
I20260812 02:31:46.167753 14748 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbep4M4/build/debug/bin/kudu
/tmp/dist-test-taskbep4M4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501898713699-14748-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501898713699-14748-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501898713699-14748-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501898713699-14748-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.103.1:0
--local_ip_for_outbound_sockets=127.14.103.1
--webserver_interface=127.14.103.1
--webserver_port=0
--tserver_master_addrs=127.14.103.62:40425
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.1.keytab
--principal=kudu/127.14.103.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.103.20:32793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501898713699-14748-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501898713699-14748-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501898713699-14748-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:31:46.301995 16054 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:31:46.302338 16054 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:31:46.302389 16054 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:31:46.306787 16054 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:31:46.306879 16054 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:31:46.306922 16054 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 02:31:46.307017 16054 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.103.1
I20260812 02:31:46.313553 16054 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.103.20:32793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501898713699-14748-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501898713699-14748-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.1.keytab
--principal=kudu/127.14.103.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.103.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501898713699-14748-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.103.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.14.103.62:40425
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16054
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.103.1
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501898713699-14748-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:31:46.314790 16054 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:31:46.315924 16054 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:31:46.326872 16062 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:31:46.327054 16054 server_base.cc:1034] running on GCE node
W20260812 02:31:46.326875 16060 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:31:46.326872 16059 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:31:46.327713 16054 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:31:46.328533 16054 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:31:46.329758 16054 hybrid_clock.cc:648] HybridClock initialized: now 1786501906329711 us; error 57 us; skew 500 ppm
Aug 12 02:31:46 dist-test-slave-2w3w krb5kdc[15928](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501906, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:31:46.333052 16054 init.cc:377] Logged in from keytab as kudu/127.14.103.1@KRBTEST.COM (short username kudu)
I20260812 02:31:46.334247 16054 webserver.cc:460] Webserver started at http://127.14.103.1:44333/ using document root <none> and password file <none>
I20260812 02:31:46.334875 16054 fs_manager.cc:359] Metadata directory not provided
I20260812 02:31:46.334949 16054 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:31:46.335239 16054 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:31:46.338320 16054 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501898713699-14748-0/minicluster-data/ts-0/data/instance:
uuid: "8f9b0cac475346b7a96b276da650a22c"
format_stamp: "Formatted at 2026-08-12 02:31:46 on dist-test-slave-2w3w"
server_key: "674e5bcafef9e2abea73fbb6ef4d15d8"
server_key_iv: "124353c29459ee24c0c09c67a987b7da"
server_key_version: "encryptionkey@0"
I20260812 02:31:46.339088 16054 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501898713699-14748-0/minicluster-data/ts-0/wal/instance:
uuid: "8f9b0cac475346b7a96b276da650a22c"
format_stamp: "Formatted at 2026-08-12 02:31:46 on dist-test-slave-2w3w"
server_key: "674e5bcafef9e2abea73fbb6ef4d15d8"
server_key_iv: "124353c29459ee24c0c09c67a987b7da"
server_key_version: "encryptionkey@0"
I20260812 02:31:46.343701 16054 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260812 02:31:46.346761 16070 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:31:46.348480 16054 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 02:31:46.348670 16054 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501898713699-14748-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501898713699-14748-0/minicluster-data/ts-0/wal
uuid: "8f9b0cac475346b7a96b276da650a22c"
format_stamp: "Formatted at 2026-08-12 02:31:46 on dist-test-slave-2w3w"
server_key: "674e5bcafef9e2abea73fbb6ef4d15d8"
server_key_iv: "124353c29459ee24c0c09c67a987b7da"
server_key_version: "encryptionkey@0"
I20260812 02:31:46.348803 16054 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501898713699-14748-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501898713699-14748-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501898713699-14748-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:31:46.385365 16054 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:31:46.389443 16054 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:31:46.389650 16054 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:31:46.391300 16054 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:31:46.391392 16054 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:31:46.391471 16054 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:31:46.391520 16054 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 12 02:31:46 dist-test-slave-2w3w krb5kdc[15928](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501906, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.1@KRBTEST.COM for kudu/127.14.103.62@KRBTEST.COM
I20260812 02:31:46.404002 16054 rpc_server.cc:307] RPC server started. Bound to: 127.14.103.1:35705
I20260812 02:31:46.404016 16192 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.103.1:35705 every 8 connection(s)
I20260812 02:31:46.405356 16054 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501898713699-14748-0/minicluster-data/ts-0/data/info.pb
I20260812 02:31:46.407260 14748 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbep4M4/build/debug/bin/kudu as pid 16054
I20260812 02:31:46.407406 14748 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501898713699-14748-0/minicluster-data/ts-0/wal/instance
I20260812 02:31:46.407701 14748 external_mini_cluster.cc:1428] Setting key 4d6471e0d4d3c881c059d19cc5673ff2
I20260812 02:31:46.408169 16030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:31:46.392394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.1:44275 (local address 127.14.103.62:40425)
0812 02:31:46.392599 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0812 02:31:46.392604 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:31:46.393476 (+ 872us) server_negotiation.cc:407] Connection header received
0812 02:31:46.394579 (+ 1103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:31:46.394586 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:31:46.394666 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:31:46.394797 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:31:46.396327 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:31:46.396917 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:31:46.398470 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:31:46.398724 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:31:46.398847 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:31:46.403113 (+ 4266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:31:46.403116 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:31:46.403118 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:31:46.403152 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:31:46.405037 (+ 1885us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:31:46.405818 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:31:46.405824 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:31:46.405826 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:31:46.405995 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:31:46.406442 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:31:46.406446 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:31:46.406448 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:31:46.406711 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:31:46.406810 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0812 02:31:46.407724 (+ 914us) server_negotiation.cc:299] Negotiation successful
0812 02:31:46.407981 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260812 02:31:46.409310 16079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:31:46.392791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:40425 (local address 127.14.103.1:44275)
0812 02:31:46.393269 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0812 02:31:46.393325 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0812 02:31:46.394310 (+ 985us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:31:46.395057 (+ 747us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:31:46.395067 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:31:46.395506 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:31:46.396146 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:31:46.396173 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:31:46.397177 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:31:46.397184 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:31:46.398318 (+ 1134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:31:46.398330 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:31:46.398917 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:31:46.398921 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:31:46.399138 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:31:46.399868 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:31:46.399887 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:31:46.402786 (+ 2899us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:31:46.405190 (+ 2404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:31:46.405197 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:31:46.405213 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:31:46.405641 (+ 428us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:31:46.406130 (+ 489us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:31:46.406133 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:31:46.406135 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:31:46.406317 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:31:46.406819 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:31:46.406824 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:31:46.407283 (+ 459us) client_negotiation.cc:769] Sending connection context
0812 02:31:46.407675 (+ 392us) client_negotiation.cc:240] Negotiation successful
0812 02:31:46.408206 (+ 531us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":145,"threads_started":1}
I20260812 02:31:46.410952 16193 heartbeater.cc:347] Connected to a master server at 127.14.103.62:40425
I20260812 02:31:46.411197 16193 heartbeater.cc:464] Registering TS with master...
I20260812 02:31:46.411906 16193 heartbeater.cc:511] Master 127.14.103.62:40425 requested a full tablet report, sending...
I20260812 02:31:46.413800 15975 ts_manager.cc:194] Registered new tserver with Master: 8f9b0cac475346b7a96b276da650a22c (127.14.103.1:35705)
I20260812 02:31:46.415735 15975 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.103.1@KRBTEST.COM'} at 127.14.103.1:44275
WARNING: no policy specified for kudu/127.14.103.2@KRBTEST.COM; defaulting to no policy
I20260812 02:31:46.424024 16200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:31:46.413614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:40425 (local address 127.14.103.1:58441)
0812 02:31:46.413958 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0812 02:31:46.413976 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 02:31:46.414097 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:31:46.414374 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:31:46.414377 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:31:46.414618 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:31:46.414728 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:31:46.414735 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:31:46.415770 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:31:46.415774 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:31:46.417092 (+ 1318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:31:46.417110 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:31:46.417838 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:31:46.417842 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:31:46.417967 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:31:46.418652 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:31:46.418673 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:31:46.419276 (+ 603us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:31:46.421699 (+ 2423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:31:46.421702 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:31:46.421703 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:31:46.421945 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:31:46.422384 (+ 439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:31:46.422386 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:31:46.422388 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:31:46.422481 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:31:46.423502 (+ 1021us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:31:46.423505 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:31:46.423614 (+ 109us) client_negotiation.cc:769] Sending connection context
0812 02:31:46.423710 (+ 96us) client_negotiation.cc:240] Negotiation successful
0812 02:31:46.423854 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":145,"threads_started":1}
I20260812 02:31:46.424212 16030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:31:46.413667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.1:58441 (local address 127.14.103.62:40425)
0812 02:31:46.413981 (+ 314us) server_negotiation.cc:206] Beginning negotiation
0812 02:31:46.413987 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 02:31:46.414040 (+ 53us) server_negotiation.cc:407] Connection header received
0812 02:31:46.414208 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:31:46.414211 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:31:46.414258 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:31:46.414357 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:31:46.414881 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:31:46.415626 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:31:46.417294 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:31:46.417670 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:31:46.417799 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:31:46.419472 (+ 1673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:31:46.419476 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:31:46.419478 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:31:46.419505 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:31:46.421561 (+ 2056us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:31:46.422118 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:31:46.422121 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:31:46.422122 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:31:46.422214 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:31:46.422636 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:31:46.422640 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:31:46.422641 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:31:46.423365 (+ 724us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:31:46.423475 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0812 02:31:46.423822 (+ 347us) server_negotiation.cc:299] Negotiation successful
0812 02:31:46.423952 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":229}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.103.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.103.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.2.keytab.
Entry for principal kudu/127.14.103.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.2.keytab.
I20260812 02:31:46.452158 14748 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbep4M4/build/debug/bin/kudu
/tmp/dist-test-taskbep4M4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501898713699-14748-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501898713699-14748-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501898713699-14748-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501898713699-14748-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.103.2:0
--local_ip_for_outbound_sockets=127.14.103.2
--webserver_interface=127.14.103.2
--webserver_port=0
--tserver_master_addrs=127.14.103.62:40425
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.2.keytab
--principal=kudu/127.14.103.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.103.20:32793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501898713699-14748-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501898713699-14748-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501898713699-14748-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:31:46.586186 16202 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:31:46.586534 16202 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:31:46.586622 16202 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:31:46.591157 16202 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:31:46.591256 16202 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:31:46.591331 16202 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 02:31:46.591413 16202 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.103.2
I20260812 02:31:46.597299 16202 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.103.20:32793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501898713699-14748-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501898713699-14748-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.2.keytab
--principal=kudu/127.14.103.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.103.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501898713699-14748-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.103.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.14.103.62:40425
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16202
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.103.2
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501898713699-14748-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:31:46.598523 16202 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:31:46.599654 16202 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:31:46.610415 16208 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:31:46.610586 16210 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:31:46.610415 16207 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:31:46.610927 16202 server_base.cc:1034] running on GCE node
I20260812 02:31:46.611498 16202 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:31:46.612242 16202 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:31:46.613457 16202 hybrid_clock.cc:648] HybridClock initialized: now 1786501906613416 us; error 54 us; skew 500 ppm
Aug 12 02:31:46 dist-test-slave-2w3w krb5kdc[15928](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501906, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:31:46.616909 16202 init.cc:377] Logged in from keytab as kudu/127.14.103.2@KRBTEST.COM (short username kudu)
I20260812 02:31:46.618139 16202 webserver.cc:460] Webserver started at http://127.14.103.2:46019/ using document root <none> and password file <none>
I20260812 02:31:46.618784 16202 fs_manager.cc:359] Metadata directory not provided
I20260812 02:31:46.618892 16202 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:31:46.619169 16202 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:31:46.621482 16202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501898713699-14748-0/minicluster-data/ts-1/data/instance:
uuid: "2bf93dc7aa034d5b8cab5efdc5dc0137"
format_stamp: "Formatted at 2026-08-12 02:31:46 on dist-test-slave-2w3w"
server_key: "fcda9204a949b6914e356bb04163f8d1"
server_key_iv: "b1a93b34dc17315d3b02fd4007fd8e21"
server_key_version: "encryptionkey@0"
I20260812 02:31:46.622117 16202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501898713699-14748-0/minicluster-data/ts-1/wal/instance:
uuid: "2bf93dc7aa034d5b8cab5efdc5dc0137"
format_stamp: "Formatted at 2026-08-12 02:31:46 on dist-test-slave-2w3w"
server_key: "fcda9204a949b6914e356bb04163f8d1"
server_key_iv: "b1a93b34dc17315d3b02fd4007fd8e21"
server_key_version: "encryptionkey@0"
I20260812 02:31:46.626441 16202 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260812 02:31:46.629582 16218 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:31:46.631333 16202 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 02:31:46.631511 16202 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501898713699-14748-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501898713699-14748-0/minicluster-data/ts-1/wal
uuid: "2bf93dc7aa034d5b8cab5efdc5dc0137"
format_stamp: "Formatted at 2026-08-12 02:31:46 on dist-test-slave-2w3w"
server_key: "fcda9204a949b6914e356bb04163f8d1"
server_key_iv: "b1a93b34dc17315d3b02fd4007fd8e21"
server_key_version: "encryptionkey@0"
I20260812 02:31:46.631654 16202 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501898713699-14748-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501898713699-14748-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501898713699-14748-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:31:46.656713 16202 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:31:46.660488 16202 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:31:46.660742 16202 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:31:46.662375 16202 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:31:46.662453 16202 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:31:46.662531 16202 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:31:46.662571 16202 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 12 02:31:46 dist-test-slave-2w3w krb5kdc[15928](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501906, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.2@KRBTEST.COM for kudu/127.14.103.62@KRBTEST.COM
I20260812 02:31:46.674386 16202 rpc_server.cc:307] RPC server started. Bound to: 127.14.103.2:33765
I20260812 02:31:46.674499 16347 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.103.2:33765 every 8 connection(s)
I20260812 02:31:46.675715 16202 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501898713699-14748-0/minicluster-data/ts-1/data/info.pb
I20260812 02:31:46.677774 16030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:31:46.663258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.2:54135 (local address 127.14.103.62:40425)
0812 02:31:46.663434 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0812 02:31:46.663439 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:31:46.664505 (+ 1066us) server_negotiation.cc:407] Connection header received
0812 02:31:46.665889 (+ 1384us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:31:46.665894 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:31:46.665967 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:31:46.666103 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:31:46.667586 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:31:46.668205 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:31:46.669384 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:31:46.669685 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:31:46.669813 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:31:46.673536 (+ 3723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:31:46.673540 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:31:46.673543 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:31:46.673579 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:31:46.675519 (+ 1940us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:31:46.676075 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:31:46.676079 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:31:46.676081 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:31:46.676188 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:31:46.676616 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:31:46.676620 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:31:46.676622 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:31:46.676845 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:31:46.676960 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0812 02:31:46.677474 (+ 514us) server_negotiation.cc:299] Negotiation successful
0812 02:31:46.677610 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260812 02:31:46.678604 16227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:31:46.663739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:40425 (local address 127.14.103.2:54135)
0812 02:31:46.664306 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0812 02:31:46.664372 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0812 02:31:46.665622 (+ 1250us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:31:46.666284 (+ 662us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:31:46.666291 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:31:46.666759 (+ 468us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:31:46.667421 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:31:46.667439 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:31:46.668334 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:31:46.668338 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:31:46.669246 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:31:46.669255 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:31:46.669900 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:31:46.669908 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:31:46.670167 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:31:46.670996 (+ 829us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:31:46.671018 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:31:46.673269 (+ 2251us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:31:46.675657 (+ 2388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:31:46.675662 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:31:46.675674 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:31:46.675957 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:31:46.676321 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:31:46.676324 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:31:46.676326 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:31:46.676526 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:31:46.676958 (+ 432us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:31:46.676965 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:31:46.677226 (+ 261us) client_negotiation.cc:769] Sending connection context
0812 02:31:46.677430 (+ 204us) client_negotiation.cc:240] Negotiation successful
0812 02:31:46.677771 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":126,"threads_started":1}
I20260812 02:31:46.680148 16348 heartbeater.cc:347] Connected to a master server at 127.14.103.62:40425
I20260812 02:31:46.680348 16348 heartbeater.cc:464] Registering TS with master...
I20260812 02:31:46.681059 16348 heartbeater.cc:511] Master 127.14.103.62:40425 requested a full tablet report, sending...
I20260812 02:31:46.681931 14748 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbep4M4/build/debug/bin/kudu as pid 16202
I20260812 02:31:46.682052 14748 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501898713699-14748-0/minicluster-data/ts-1/wal/instance
I20260812 02:31:46.682281 15975 ts_manager.cc:194] Registered new tserver with Master: 2bf93dc7aa034d5b8cab5efdc5dc0137 (127.14.103.2:33765)
I20260812 02:31:46.682375 14748 external_mini_cluster.cc:1428] Setting key d6f0b82e83639cbb641f419a6b49d2fb
I20260812 02:31:46.683049 15975 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.103.2@KRBTEST.COM'} at 127.14.103.2:54135
I20260812 02:31:46.691247 16030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:31:46.682499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.2:50143 (local address 127.14.103.62:40425)
0812 02:31:46.682635 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0812 02:31:46.682640 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:31:46.683127 (+ 487us) server_negotiation.cc:407] Connection header received
0812 02:31:46.683245 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:31:46.683248 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:31:46.683295 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:31:46.683401 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:31:46.684150 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:31:46.684721 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:31:46.685712 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:31:46.685976 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:31:46.686100 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:31:46.687449 (+ 1349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:31:46.687453 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:31:46.687456 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:31:46.687486 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:31:46.689224 (+ 1738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:31:46.689737 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:31:46.689741 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:31:46.689743 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:31:46.689840 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:31:46.690151 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:31:46.690156 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:31:46.690158 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:31:46.690436 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:31:46.690567 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0812 02:31:46.690883 (+ 316us) server_negotiation.cc:299] Negotiation successful
0812 02:31:46.691064 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260812 02:31:46.691262 16355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:31:46.682437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:40425 (local address 127.14.103.2:50143)
0812 02:31:46.683028 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0812 02:31:46.683046 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 02:31:46.683159 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:31:46.683418 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:31:46.683422 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:31:46.683798 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:31:46.683979 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:31:46.683992 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:31:46.684878 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:31:46.684882 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:31:46.685566 (+ 684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:31:46.685576 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:31:46.686108 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:31:46.686112 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:31:46.686216 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:31:46.686833 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:31:46.686846 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:31:46.687290 (+ 444us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:31:46.689373 (+ 2083us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:31:46.689377 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:31:46.689380 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:31:46.689619 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:31:46.689944 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:31:46.689948 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:31:46.689950 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:31:46.690049 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:31:46.690553 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:31:46.690556 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:31:46.690688 (+ 132us) client_negotiation.cc:769] Sending connection context
0812 02:31:46.690806 (+ 118us) client_negotiation.cc:240] Negotiation successful
0812 02:31:46.691053 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":113,"threads_started":1}
WARNING: no policy specified for kudu/127.14.103.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.103.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.103.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.3.keytab.
Entry for principal kudu/127.14.103.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.3.keytab.
I20260812 02:31:46.719859 14748 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbep4M4/build/debug/bin/kudu
/tmp/dist-test-taskbep4M4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501898713699-14748-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501898713699-14748-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501898713699-14748-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501898713699-14748-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.103.3:0
--local_ip_for_outbound_sockets=127.14.103.3
--webserver_interface=127.14.103.3
--webserver_port=0
--tserver_master_addrs=127.14.103.62:40425
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.3.keytab
--principal=kudu/127.14.103.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.103.20:32793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501898713699-14748-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501898713699-14748-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501898713699-14748-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:31:46.850486 16358 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:31:46.850859 16358 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:31:46.850924 16358 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:31:46.856600 16358 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:31:46.856714 16358 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:31:46.856761 16358 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 02:31:46.856858 16358 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.103.3
I20260812 02:31:46.865018 16358 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.103.20:32793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501898713699-14748-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501898713699-14748-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.3.keytab
--principal=kudu/127.14.103.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.103.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501898713699-14748-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.103.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.14.103.62:40425
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16358
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.103.3
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501898713699-14748-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:31:46.866309 16358 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:31:46.867558 16358 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:31:46.878895 16364 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:31:46.878886 16367 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:31:46.878886 16365 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:31:46.879262 16358 server_base.cc:1034] running on GCE node
I20260812 02:31:46.879880 16358 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:31:46.880832 16358 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:31:46.882032 16358 hybrid_clock.cc:648] HybridClock initialized: now 1786501906882001 us; error 47 us; skew 500 ppm
Aug 12 02:31:46 dist-test-slave-2w3w krb5kdc[15928](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501906, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:31:46.885881 16358 init.cc:377] Logged in from keytab as kudu/127.14.103.3@KRBTEST.COM (short username kudu)
I20260812 02:31:46.886904 16358 webserver.cc:460] Webserver started at http://127.14.103.3:38341/ using document root <none> and password file <none>
I20260812 02:31:46.887646 16358 fs_manager.cc:359] Metadata directory not provided
I20260812 02:31:46.887718 16358 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:31:46.887967 16358 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:31:46.890311 16358 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501898713699-14748-0/minicluster-data/ts-2/data/instance:
uuid: "e3424a0a1ba94a0aa60efc89766f5e8e"
format_stamp: "Formatted at 2026-08-12 02:31:46 on dist-test-slave-2w3w"
server_key: "fa52f7803b4409f0295b56292aaaefda"
server_key_iv: "9f135b2a5e8680075efe2aa3672b43e7"
server_key_version: "encryptionkey@0"
I20260812 02:31:46.891067 16358 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501898713699-14748-0/minicluster-data/ts-2/wal/instance:
uuid: "e3424a0a1ba94a0aa60efc89766f5e8e"
format_stamp: "Formatted at 2026-08-12 02:31:46 on dist-test-slave-2w3w"
server_key: "fa52f7803b4409f0295b56292aaaefda"
server_key_iv: "9f135b2a5e8680075efe2aa3672b43e7"
server_key_version: "encryptionkey@0"
I20260812 02:31:46.895841 16358 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260812 02:31:46.899207 16374 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:31:46.900665 16358 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 02:31:46.900856 16358 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501898713699-14748-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501898713699-14748-0/minicluster-data/ts-2/wal
uuid: "e3424a0a1ba94a0aa60efc89766f5e8e"
format_stamp: "Formatted at 2026-08-12 02:31:46 on dist-test-slave-2w3w"
server_key: "fa52f7803b4409f0295b56292aaaefda"
server_key_iv: "9f135b2a5e8680075efe2aa3672b43e7"
server_key_version: "encryptionkey@0"
I20260812 02:31:46.901005 16358 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501898713699-14748-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501898713699-14748-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501898713699-14748-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:31:46.931639 16358 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:31:46.937633 16358 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:31:46.938067 16358 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:31:46.940416 16358 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:31:46.940502 16358 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:31:46.940569 16358 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:31:46.940618 16358 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:31:46.952716 16358 rpc_server.cc:307] RPC server started. Bound to: 127.14.103.3:37431
I20260812 02:31:46.952988 16500 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.103.3:37431 every 8 connection(s)
I20260812 02:31:46.954028 16358 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501898713699-14748-0/minicluster-data/ts-2/data/info.pb
Aug 12 02:31:46 dist-test-slave-2w3w krb5kdc[15928](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501906, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.3@KRBTEST.COM for kudu/127.14.103.62@KRBTEST.COM
I20260812 02:31:46.959863 14748 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbep4M4/build/debug/bin/kudu as pid 16358
I20260812 02:31:46.960040 14748 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501898713699-14748-0/minicluster-data/ts-2/wal/instance
I20260812 02:31:46.960427 14748 external_mini_cluster.cc:1428] Setting key d078ddaa116e23da03717c030080c5f0
I20260812 02:31:46.963068 16030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:31:46.941868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.3:58839 (local address 127.14.103.62:40425)
0812 02:31:46.942036 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0812 02:31:46.942041 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:31:46.943894 (+ 1853us) server_negotiation.cc:407] Connection header received
0812 02:31:46.945991 (+ 2097us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:31:46.945996 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:31:46.946077 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:31:46.946192 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:31:46.948515 (+ 2323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:31:46.949542 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:31:46.952518 (+ 2976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:31:46.952879 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:31:46.953027 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:31:46.957340 (+ 4313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:31:46.957344 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:31:46.957346 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:31:46.957379 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:31:46.959534 (+ 2155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:31:46.960725 (+ 1191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:31:46.960731 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:31:46.960733 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:31:46.960899 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:31:46.961690 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:31:46.961694 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:31:46.961696 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:31:46.961956 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:31:46.962078 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0812 02:31:46.962683 (+ 605us) server_negotiation.cc:299] Negotiation successful
0812 02:31:46.962859 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260812 02:31:46.964057 16384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:31:46.942358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:40425 (local address 127.14.103.3:58839)
0812 02:31:46.943568 (+ 1210us) negotiation.cc:107] Waiting for socket to connect
0812 02:31:46.943628 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0812 02:31:46.945643 (+ 2015us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:31:46.946407 (+ 764us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:31:46.946416 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:31:46.946906 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:31:46.948098 (+ 1192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:31:46.948133 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:31:46.949854 (+ 1721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:31:46.949859 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:31:46.952263 (+ 2404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:31:46.952287 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:31:46.953067 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:31:46.953073 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:31:46.953295 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:31:46.954453 (+ 1158us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:31:46.954482 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:31:46.957005 (+ 2523us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:31:46.959804 (+ 2799us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:31:46.959811 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:31:46.959829 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:31:46.960525 (+ 696us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:31:46.961095 (+ 570us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:31:46.961101 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:31:46.961103 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:31:46.961535 (+ 432us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:31:46.962094 (+ 559us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:31:46.962103 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:31:46.962409 (+ 306us) client_negotiation.cc:769] Sending connection context
0812 02:31:46.962614 (+ 205us) client_negotiation.cc:240] Negotiation successful
0812 02:31:46.963040 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":938,"thread_start_us":698,"threads_started":1}
I20260812 02:31:46.965554 16501 heartbeater.cc:347] Connected to a master server at 127.14.103.62:40425
I20260812 02:31:46.965746 16501 heartbeater.cc:464] Registering TS with master...
I20260812 02:31:46.966455 16501 heartbeater.cc:511] Master 127.14.103.62:40425 requested a full tablet report, sending...
I20260812 02:31:46.967911 15976 ts_manager.cc:194] Registered new tserver with Master: e3424a0a1ba94a0aa60efc89766f5e8e (127.14.103.3:37431)
I20260812 02:31:46.968668 15976 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.103.3@KRBTEST.COM'} at 127.14.103.3:58839
I20260812 02:31:46.977057 14748 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 02:31:46.978284 16030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:31:46.968566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.3:51619 (local address 127.14.103.62:40425)
0812 02:31:46.968717 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0812 02:31:46.968723 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 02:31:46.969228 (+ 505us) server_negotiation.cc:407] Connection header received
0812 02:31:46.969278 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:31:46.969281 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:31:46.969322 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:31:46.969431 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:31:46.970015 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:31:46.970572 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:31:46.971501 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:31:46.971724 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:31:46.971842 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:31:46.973249 (+ 1407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:31:46.973253 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:31:46.973255 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:31:46.973282 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:31:46.975812 (+ 2530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:31:46.976688 (+ 876us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:31:46.976696 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:31:46.976701 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:31:46.976834 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:31:46.977270 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:31:46.977273 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:31:46.977274 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:31:46.977508 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:31:46.977635 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0812 02:31:46.977949 (+ 314us) server_negotiation.cc:299] Negotiation successful
0812 02:31:46.978100 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260812 02:31:46.978304 16508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:31:46.968445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:40425 (local address 127.14.103.3:51619)
0812 02:31:46.969020 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0812 02:31:46.969040 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 02:31:46.969174 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:31:46.969500 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:31:46.969503 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:31:46.969739 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:31:46.969865 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:31:46.969871 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:31:46.970703 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:31:46.970706 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:31:46.971370 (+ 664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:31:46.971380 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:31:46.971844 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:31:46.971848 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:31:46.971947 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:31:46.972642 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:31:46.972657 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:31:46.973062 (+ 405us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:31:46.976122 (+ 3060us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:31:46.976125 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:31:46.976128 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:31:46.976472 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:31:46.976964 (+ 492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:31:46.976968 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:31:46.976970 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:31:46.977080 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:31:46.977621 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:31:46.977624 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:31:46.977745 (+ 121us) client_negotiation.cc:769] Sending connection context
0812 02:31:46.977870 (+ 125us) client_negotiation.cc:240] Negotiation successful
0812 02:31:46.978081 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":111,"threads_started":1}
Aug 12 02:31:46 dist-test-slave-2w3w krb5kdc[15928](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501906, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 12 02:31:46 dist-test-slave-2w3w krb5kdc[15928](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501906, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.14.103.62@KRBTEST.COM
I20260812 02:31:47.012202 16030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:31:47.000323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32902 (local address 127.14.103.62:40425)
0812 02:31:47.000459 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0812 02:31:47.000463 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:31:47.000476 (+ 13us) server_negotiation.cc:407] Connection header received
0812 02:31:47.000520 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:31:47.000522 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:31:47.000566 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:31:47.000644 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:31:47.001976 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:31:47.002550 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:31:47.003574 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:31:47.003803 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:31:47.003941 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:31:47.007381 (+ 3440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:31:47.007385 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:31:47.007387 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:31:47.007427 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:31:47.009245 (+ 1818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:31:47.009967 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:31:47.009971 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:31:47.009972 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:31:47.010079 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:31:47.010574 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:31:47.010577 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:31:47.010578 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:31:47.011298 (+ 720us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:31:47.011433 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0812 02:31:47.011923 (+ 490us) server_negotiation.cc:299] Negotiation successful
0812 02:31:47.012065 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260812 02:31:47.017699 15976 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:32902:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 02:31:47.021389 15976 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 02:31:47.040192 16520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:31:47.033333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.3:37431 (local address 127.0.0.1:48742)
0812 02:31:47.033990 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0812 02:31:47.034006 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 02:31:47.034126 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:31:47.035124 (+ 998us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:31:47.035127 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:31:47.035146 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:31:47.035234 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:31:47.035242 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:31:47.036587 (+ 1345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:31:47.036590 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:31:47.038543 (+ 1953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:31:47.038551 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:31:47.039546 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:31:47.039550 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:31:47.039678 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:31:47.039728 (+ 50us) client_negotiation.cc:769] Sending connection context
0812 02:31:47.039851 (+ 123us) client_negotiation.cc:240] Negotiation successful
0812 02:31:47.040018 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":593,"mutex_wait_us":1050,"thread_start_us":176,"threads_started":1}
I20260812 02:31:47.040341 16521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:31:47.032338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47842 (local address 127.14.103.1:35705)
0812 02:31:47.033137 (+ 799us) server_negotiation.cc:206] Beginning negotiation
0812 02:31:47.033145 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 02:31:47.033796 (+ 651us) server_negotiation.cc:407] Connection header received
0812 02:31:47.033890 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:31:47.033897 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:31:47.034120 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:31:47.034268 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:31:47.034615 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:31:47.036022 (+ 1407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:31:47.038215 (+ 2193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:31:47.039536 (+ 1321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:31:47.039662 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:31:47.039816 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0812 02:31:47.040059 (+ 243us) server_negotiation.cc:299] Negotiation successful
0812 02:31:47.040186 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":680,"thread_start_us":121,"threads_started":1}
I20260812 02:31:47.040853 16524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:31:47.033154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51176 (local address 127.14.103.2:33765)
0812 02:31:47.033596 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0812 02:31:47.033608 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 02:31:47.033859 (+ 251us) server_negotiation.cc:407] Connection header received
0812 02:31:47.034050 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:31:47.034055 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:31:47.034211 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:31:47.034314 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:31:47.034754 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:31:47.035983 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:31:47.038214 (+ 2231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:31:47.039525 (+ 1311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:31:47.039662 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:31:47.039854 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0812 02:31:47.040537 (+ 683us) server_negotiation.cc:299] Negotiation successful
0812 02:31:47.040660 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":131,"threads_started":1}
I20260812 02:31:47.040416 16525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:31:47.032398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.2:33765 (local address 127.0.0.1:51176)
0812 02:31:47.033754 (+ 1356us) negotiation.cc:107] Waiting for socket to connect
0812 02:31:47.033769 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 02:31:47.033923 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:31:47.034482 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:31:47.034486 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:31:47.034506 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:31:47.034600 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:31:47.034607 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:31:47.036311 (+ 1704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:31:47.036314 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:31:47.037827 (+ 1513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:31:47.037841 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:31:47.039934 (+ 2093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:31:47.039938 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:31:47.040048 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:31:47.040080 (+ 32us) client_negotiation.cc:769] Sending connection context
0812 02:31:47.040163 (+ 83us) client_negotiation.cc:240] Negotiation successful
0812 02:31:47.040258 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1286,"mutex_wait_us":8,"thread_start_us":109,"threads_started":1}
I20260812 02:31:47.040798 16523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:31:47.032205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.1:35705 (local address 127.0.0.1:47842)
0812 02:31:47.033427 (+ 1222us) negotiation.cc:107] Waiting for socket to connect
0812 02:31:47.033489 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0812 02:31:47.033644 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:31:47.034342 (+ 698us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:31:47.034347 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:31:47.034367 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:31:47.034488 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:31:47.034493 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:31:47.036264 (+ 1771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:31:47.036270 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:31:47.037989 (+ 1719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:31:47.037997 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:31:47.039775 (+ 1778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:31:47.039778 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:31:47.039883 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:31:47.039905 (+ 22us) client_negotiation.cc:769] Sending connection context
0812 02:31:47.040556 (+ 651us) client_negotiation.cc:240] Negotiation successful
0812 02:31:47.040656 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1011,"mutex_wait_us":441,"thread_start_us":183,"threads_started":1}
I20260812 02:31:47.041441 16526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:31:47.033974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48742 (local address 127.14.103.3:37431)
0812 02:31:47.034591 (+ 617us) server_negotiation.cc:206] Beginning negotiation
0812 02:31:47.034603 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 02:31:47.034625 (+ 22us) server_negotiation.cc:407] Connection header received
0812 02:31:47.034701 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:31:47.034705 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:31:47.034902 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:31:47.035146 (+ 244us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:31:47.035364 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:31:47.036465 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:31:47.038689 (+ 2224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:31:47.039392 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:31:47.039521 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:31:47.039713 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0812 02:31:47.040707 (+ 994us) server_negotiation.cc:299] Negotiation successful
0812 02:31:47.040818 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":521,"thread_start_us":170,"threads_started":1}
I20260812 02:31:47.044260 16121 tablet_service.cc:1467] Processing CreateTablet for tablet b3c7bbc8c22442d5b46722402a0a1bea (DEFAULT_TABLE table=test-table [id=8683a37b764b46cfbf5650c10c16e0fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:31:47.045135 16274 tablet_service.cc:1467] Processing CreateTablet for tablet b3c7bbc8c22442d5b46722402a0a1bea (DEFAULT_TABLE table=test-table [id=8683a37b764b46cfbf5650c10c16e0fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:31:47.045651 16121 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3c7bbc8c22442d5b46722402a0a1bea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:31:47.045331 16428 tablet_service.cc:1467] Processing CreateTablet for tablet b3c7bbc8c22442d5b46722402a0a1bea (DEFAULT_TABLE table=test-table [id=8683a37b764b46cfbf5650c10c16e0fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:31:47.046346 16274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3c7bbc8c22442d5b46722402a0a1bea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:31:47.046900 16428 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3c7bbc8c22442d5b46722402a0a1bea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:31:47.058805 16527 tablet_bootstrap.cc:492] T b3c7bbc8c22442d5b46722402a0a1bea P e3424a0a1ba94a0aa60efc89766f5e8e: Bootstrap starting.
I20260812 02:31:47.061389 16528 tablet_bootstrap.cc:492] T b3c7bbc8c22442d5b46722402a0a1bea P 2bf93dc7aa034d5b8cab5efdc5dc0137: Bootstrap starting.
I20260812 02:31:47.063736 16527 tablet_bootstrap.cc:654] T b3c7bbc8c22442d5b46722402a0a1bea P e3424a0a1ba94a0aa60efc89766f5e8e: Neither blocks nor log segments found. Creating new log.
I20260812 02:31:47.063743 16528 tablet_bootstrap.cc:654] T b3c7bbc8c22442d5b46722402a0a1bea P 2bf93dc7aa034d5b8cab5efdc5dc0137: Neither blocks nor log segments found. Creating new log.
I20260812 02:31:47.064618 16527 log.cc:824] T b3c7bbc8c22442d5b46722402a0a1bea P e3424a0a1ba94a0aa60efc89766f5e8e: Log is configured to *not* fsync() on all Append() calls
I20260812 02:31:47.064657 16528 log.cc:824] T b3c7bbc8c22442d5b46722402a0a1bea P 2bf93dc7aa034d5b8cab5efdc5dc0137: Log is configured to *not* fsync() on all Append() calls
I20260812 02:31:47.067111 16527 tablet_bootstrap.cc:492] T b3c7bbc8c22442d5b46722402a0a1bea P e3424a0a1ba94a0aa60efc89766f5e8e: No bootstrap required, opened a new log
I20260812 02:31:47.067456 16527 ts_tablet_manager.cc:1397] T b3c7bbc8c22442d5b46722402a0a1bea P e3424a0a1ba94a0aa60efc89766f5e8e: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260812 02:31:47.069114 16528 tablet_bootstrap.cc:492] T b3c7bbc8c22442d5b46722402a0a1bea P 2bf93dc7aa034d5b8cab5efdc5dc0137: No bootstrap required, opened a new log
I20260812 02:31:47.069505 16528 ts_tablet_manager.cc:1397] T b3c7bbc8c22442d5b46722402a0a1bea P 2bf93dc7aa034d5b8cab5efdc5dc0137: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260812 02:31:47.071560 16531 tablet_bootstrap.cc:492] T b3c7bbc8c22442d5b46722402a0a1bea P 8f9b0cac475346b7a96b276da650a22c: Bootstrap starting.
I20260812 02:31:47.072362 16527 raft_consensus.cc:348] T b3c7bbc8c22442d5b46722402a0a1bea P e3424a0a1ba94a0aa60efc89766f5e8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9b0cac475346b7a96b276da650a22c" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 35705 } } peers { permanent_uuid: "2bf93dc7aa034d5b8cab5efdc5dc0137" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 33765 } } peers { permanent_uuid: "e3424a0a1ba94a0aa60efc89766f5e8e" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 37431 } }
I20260812 02:31:47.072701 16527 raft_consensus.cc:374] T b3c7bbc8c22442d5b46722402a0a1bea P e3424a0a1ba94a0aa60efc89766f5e8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:31:47.072777 16527 raft_consensus.cc:729] T b3c7bbc8c22442d5b46722402a0a1bea P e3424a0a1ba94a0aa60efc89766f5e8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3424a0a1ba94a0aa60efc89766f5e8e, State: Initialized, Role: FOLLOWER
I20260812 02:31:47.073298 16527 consensus_queue.cc:260] T b3c7bbc8c22442d5b46722402a0a1bea P e3424a0a1ba94a0aa60efc89766f5e8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9b0cac475346b7a96b276da650a22c" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 35705 } } peers { permanent_uuid: "2bf93dc7aa034d5b8cab5efdc5dc0137" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 33765 } } peers { permanent_uuid: "e3424a0a1ba94a0aa60efc89766f5e8e" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 37431 } }
I20260812 02:31:47.074097 16501 heartbeater.cc:503] Master 127.14.103.62:40425 was elected leader, sending a full tablet report...
I20260812 02:31:47.074362 16531 tablet_bootstrap.cc:654] T b3c7bbc8c22442d5b46722402a0a1bea P 8f9b0cac475346b7a96b276da650a22c: Neither blocks nor log segments found. Creating new log.
I20260812 02:31:47.074569 16527 ts_tablet_manager.cc:1428] T b3c7bbc8c22442d5b46722402a0a1bea P e3424a0a1ba94a0aa60efc89766f5e8e: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260812 02:31:47.073905 16528 raft_consensus.cc:348] T b3c7bbc8c22442d5b46722402a0a1bea P 2bf93dc7aa034d5b8cab5efdc5dc0137 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9b0cac475346b7a96b276da650a22c" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 35705 } } peers { permanent_uuid: "2bf93dc7aa034d5b8cab5efdc5dc0137" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 33765 } } peers { permanent_uuid: "e3424a0a1ba94a0aa60efc89766f5e8e" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 37431 } }
I20260812 02:31:47.074901 16528 raft_consensus.cc:374] T b3c7bbc8c22442d5b46722402a0a1bea P 2bf93dc7aa034d5b8cab5efdc5dc0137 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:31:47.075003 16528 raft_consensus.cc:729] T b3c7bbc8c22442d5b46722402a0a1bea P 2bf93dc7aa034d5b8cab5efdc5dc0137 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2bf93dc7aa034d5b8cab5efdc5dc0137, State: Initialized, Role: FOLLOWER
I20260812 02:31:47.075453 16531 log.cc:824] T b3c7bbc8c22442d5b46722402a0a1bea P 8f9b0cac475346b7a96b276da650a22c: Log is configured to *not* fsync() on all Append() calls
I20260812 02:31:47.075491 16528 consensus_queue.cc:260] T b3c7bbc8c22442d5b46722402a0a1bea P 2bf93dc7aa034d5b8cab5efdc5dc0137 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9b0cac475346b7a96b276da650a22c" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 35705 } } peers { permanent_uuid: "2bf93dc7aa034d5b8cab5efdc5dc0137" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 33765 } } peers { permanent_uuid: "e3424a0a1ba94a0aa60efc89766f5e8e" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 37431 } }
I20260812 02:31:47.076745 16528 ts_tablet_manager.cc:1428] T b3c7bbc8c22442d5b46722402a0a1bea P 2bf93dc7aa034d5b8cab5efdc5dc0137: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260812 02:31:47.076802 16348 heartbeater.cc:503] Master 127.14.103.62:40425 was elected leader, sending a full tablet report...
I20260812 02:31:47.078173 16531 tablet_bootstrap.cc:492] T b3c7bbc8c22442d5b46722402a0a1bea P 8f9b0cac475346b7a96b276da650a22c: No bootstrap required, opened a new log
I20260812 02:31:47.078402 16531 ts_tablet_manager.cc:1397] T b3c7bbc8c22442d5b46722402a0a1bea P 8f9b0cac475346b7a96b276da650a22c: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260812 02:31:47.083781 16531 raft_consensus.cc:348] T b3c7bbc8c22442d5b46722402a0a1bea P 8f9b0cac475346b7a96b276da650a22c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9b0cac475346b7a96b276da650a22c" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 35705 } } peers { permanent_uuid: "2bf93dc7aa034d5b8cab5efdc5dc0137" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 33765 } } peers { permanent_uuid: "e3424a0a1ba94a0aa60efc89766f5e8e" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 37431 } }
I20260812 02:31:47.084095 16531 raft_consensus.cc:374] T b3c7bbc8c22442d5b46722402a0a1bea P 8f9b0cac475346b7a96b276da650a22c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:31:47.084182 16531 raft_consensus.cc:729] T b3c7bbc8c22442d5b46722402a0a1bea P 8f9b0cac475346b7a96b276da650a22c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f9b0cac475346b7a96b276da650a22c, State: Initialized, Role: FOLLOWER
I20260812 02:31:47.084649 16531 consensus_queue.cc:260] T b3c7bbc8c22442d5b46722402a0a1bea P 8f9b0cac475346b7a96b276da650a22c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9b0cac475346b7a96b276da650a22c" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 35705 } } peers { permanent_uuid: "2bf93dc7aa034d5b8cab5efdc5dc0137" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 33765 } } peers { permanent_uuid: "e3424a0a1ba94a0aa60efc89766f5e8e" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 37431 } }
I20260812 02:31:47.085381 16193 heartbeater.cc:503] Master 127.14.103.62:40425 was elected leader, sending a full tablet report...
I20260812 02:31:47.085646 16531 ts_tablet_manager.cc:1428] T b3c7bbc8c22442d5b46722402a0a1bea P 8f9b0cac475346b7a96b276da650a22c: Time spent starting tablet: real 0.007s user 0.003s sys 0.004s
W20260812 02:31:47.157191 16194 tablet.cc:2369] T b3c7bbc8c22442d5b46722402a0a1bea P 8f9b0cac475346b7a96b276da650a22c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 02:31:47.177245 16350 tablet.cc:2369] T b3c7bbc8c22442d5b46722402a0a1bea P 2bf93dc7aa034d5b8cab5efdc5dc0137: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 02:31:47.206327 16502 tablet.cc:2369] T b3c7bbc8c22442d5b46722402a0a1bea P e3424a0a1ba94a0aa60efc89766f5e8e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 02:31:47.337769 16535 raft_consensus.cc:482] T b3c7bbc8c22442d5b46722402a0a1bea P 8f9b0cac475346b7a96b276da650a22c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 02:31:47.338064 16535 raft_consensus.cc:504] T b3c7bbc8c22442d5b46722402a0a1bea P 8f9b0cac475346b7a96b276da650a22c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9b0cac475346b7a96b276da650a22c" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 35705 } } peers { permanent_uuid: "2bf93dc7aa034d5b8cab5efdc5dc0137" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 33765 } } peers { permanent_uuid: "e3424a0a1ba94a0aa60efc89766f5e8e" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 37431 } }
I20260812 02:31:47.339453 16535 leader_election.cc:288] T b3c7bbc8c22442d5b46722402a0a1bea P 8f9b0cac475346b7a96b276da650a22c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2bf93dc7aa034d5b8cab5efdc5dc0137 (127.14.103.2:33765), e3424a0a1ba94a0aa60efc89766f5e8e (127.14.103.3:37431)
I20260812 02:31:47.344013 16526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:31:47.339923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.1:49503 (local address 127.14.103.3:37431)
0812 02:31:47.340086 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0812 02:31:47.340093 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 02:31:47.340320 (+ 227us) server_negotiation.cc:407] Connection header received
0812 02:31:47.340671 (+ 351us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:31:47.340675 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:31:47.340737 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:31:47.340839 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:31:47.341135 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:31:47.341673 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:31:47.342775 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:31:47.343348 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:31:47.343525 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:31:47.343589 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0812 02:31:47.343787 (+ 198us) server_negotiation.cc:299] Negotiation successful
0812 02:31:47.343864 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260812 02:31:47.344029 16536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:31:47.339782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.3:37431 (local address 127.14.103.1:49503)
0812 02:31:47.340218 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0812 02:31:47.340231 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 02:31:47.340386 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:31:47.340856 (+ 470us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:31:47.340859 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:31:47.340876 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:31:47.341018 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:31:47.341025 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:31:47.341790 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:31:47.341794 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:31:47.342647 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:31:47.342656 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:31:47.343520 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:31:47.343524 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:31:47.343640 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:31:47.343671 (+ 31us) client_negotiation.cc:769] Sending connection context
0812 02:31:47.343764 (+ 93us) client_negotiation.cc:240] Negotiation successful
0812 02:31:47.343845 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":139,"threads_started":1}
I20260812 02:31:47.344239 16524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:31:47.339885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.1:60899 (local address 127.14.103.2:33765)
0812 02:31:47.340040 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0812 02:31:47.340045 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:31:47.340548 (+ 503us) server_negotiation.cc:407] Connection header received
0812 02:31:47.340742 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:31:47.340745 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:31:47.340996 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:31:47.341246 (+ 250us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:31:47.341541 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:31:47.342059 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:31:47.343218 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:31:47.343645 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:31:47.343780 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:31:47.343884 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0812 02:31:47.344033 (+ 149us) server_negotiation.cc:299] Negotiation successful
0812 02:31:47.344109 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260812 02:31:47.344272 16537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:31:47.339782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.2:33765 (local address 127.14.103.1:60899)
0812 02:31:47.340386 (+ 604us) negotiation.cc:107] Waiting for socket to connect
0812 02:31:47.340405 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 02:31:47.340580 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:31:47.341262 (+ 682us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:31:47.341265 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:31:47.341282 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:31:47.341400 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:31:47.341405 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:31:47.342172 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:31:47.342178 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:31:47.343028 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:31:47.343040 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:31:47.343772 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:31:47.343776 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:31:47.343888 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:31:47.343919 (+ 31us) client_negotiation.cc:769] Sending connection context
0812 02:31:47.344016 (+ 97us) client_negotiation.cc:240] Negotiation successful
0812 02:31:47.344089 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":453,"mutex_wait_us":37,"thread_start_us":113,"threads_started":1}
I20260812 02:31:47.344887 16449 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3c7bbc8c22442d5b46722402a0a1bea" candidate_uuid: "8f9b0cac475346b7a96b276da650a22c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3424a0a1ba94a0aa60efc89766f5e8e" is_pre_election: true
I20260812 02:31:47.344931 16297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3c7bbc8c22442d5b46722402a0a1bea" candidate_uuid: "8f9b0cac475346b7a96b276da650a22c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2bf93dc7aa034d5b8cab5efdc5dc0137" is_pre_election: true
I20260812 02:31:47.345247 16297 raft_consensus.cc:2454] T b3c7bbc8c22442d5b46722402a0a1bea P 2bf93dc7aa034d5b8cab5efdc5dc0137 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f9b0cac475346b7a96b276da650a22c in term 0.
I20260812 02:31:47.345247 16449 raft_consensus.cc:2454] T b3c7bbc8c22442d5b46722402a0a1bea P e3424a0a1ba94a0aa60efc89766f5e8e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f9b0cac475346b7a96b276da650a22c in term 0.
I20260812 02:31:47.345839 16072 leader_election.cc:302] T b3c7bbc8c22442d5b46722402a0a1bea P 8f9b0cac475346b7a96b276da650a22c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2bf93dc7aa034d5b8cab5efdc5dc0137, 8f9b0cac475346b7a96b276da650a22c, e3424a0a1ba94a0aa60efc89766f5e8e; no voters:
I20260812 02:31:47.346179 16535 raft_consensus.cc:2781] T b3c7bbc8c22442d5b46722402a0a1bea P 8f9b0cac475346b7a96b276da650a22c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 02:31:47.346274 16535 raft_consensus.cc:482] T b3c7bbc8c22442d5b46722402a0a1bea P 8f9b0cac475346b7a96b276da650a22c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 02:31:47.346307 16535 raft_consensus.cc:3037] T b3c7bbc8c22442d5b46722402a0a1bea P 8f9b0cac475346b7a96b276da650a22c [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:31:47.347623 16535 raft_consensus.cc:504] T b3c7bbc8c22442d5b46722402a0a1bea P 8f9b0cac475346b7a96b276da650a22c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9b0cac475346b7a96b276da650a22c" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 35705 } } peers { permanent_uuid: "2bf93dc7aa034d5b8cab5efdc5dc0137" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 33765 } } peers { permanent_uuid: "e3424a0a1ba94a0aa60efc89766f5e8e" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 37431 } }
I20260812 02:31:47.348034 16535 leader_election.cc:288] T b3c7bbc8c22442d5b46722402a0a1bea P 8f9b0cac475346b7a96b276da650a22c [CANDIDATE]: Term 1 election: Requested vote from peers 2bf93dc7aa034d5b8cab5efdc5dc0137 (127.14.103.2:33765), e3424a0a1ba94a0aa60efc89766f5e8e (127.14.103.3:37431)
I20260812 02:31:47.348601 16297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3c7bbc8c22442d5b46722402a0a1bea" candidate_uuid: "8f9b0cac475346b7a96b276da650a22c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2bf93dc7aa034d5b8cab5efdc5dc0137"
I20260812 02:31:47.348603 16449 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3c7bbc8c22442d5b46722402a0a1bea" candidate_uuid: "8f9b0cac475346b7a96b276da650a22c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3424a0a1ba94a0aa60efc89766f5e8e"
I20260812 02:31:47.348773 16449 raft_consensus.cc:3037] T b3c7bbc8c22442d5b46722402a0a1bea P e3424a0a1ba94a0aa60efc89766f5e8e [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:31:47.348773 16297 raft_consensus.cc:3037] T b3c7bbc8c22442d5b46722402a0a1bea P 2bf93dc7aa034d5b8cab5efdc5dc0137 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:31:47.350469 16297 raft_consensus.cc:2454] T b3c7bbc8c22442d5b46722402a0a1bea P 2bf93dc7aa034d5b8cab5efdc5dc0137 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f9b0cac475346b7a96b276da650a22c in term 1.
I20260812 02:31:47.351028 16074 leader_election.cc:302] T b3c7bbc8c22442d5b46722402a0a1bea P 8f9b0cac475346b7a96b276da650a22c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2bf93dc7aa034d5b8cab5efdc5dc0137, 8f9b0cac475346b7a96b276da650a22c; no voters:
I20260812 02:31:47.351187 16449 raft_consensus.cc:2454] T b3c7bbc8c22442d5b46722402a0a1bea P e3424a0a1ba94a0aa60efc89766f5e8e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f9b0cac475346b7a96b276da650a22c in term 1.
I20260812 02:31:47.351294 16535 raft_consensus.cc:2781] T b3c7bbc8c22442d5b46722402a0a1bea P 8f9b0cac475346b7a96b276da650a22c [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:31:47.351586 16535 raft_consensus.cc:686] T b3c7bbc8c22442d5b46722402a0a1bea P 8f9b0cac475346b7a96b276da650a22c [term 1 LEADER]: Becoming Leader. State: Replica: 8f9b0cac475346b7a96b276da650a22c, State: Running, Role: LEADER
I20260812 02:31:47.351969 16535 consensus_queue.cc:237] T b3c7bbc8c22442d5b46722402a0a1bea P 8f9b0cac475346b7a96b276da650a22c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9b0cac475346b7a96b276da650a22c" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 35705 } } peers { permanent_uuid: "2bf93dc7aa034d5b8cab5efdc5dc0137" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 33765 } } peers { permanent_uuid: "e3424a0a1ba94a0aa60efc89766f5e8e" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 37431 } }
I20260812 02:31:47.355688 15976 catalog_manager.cc:5495] T b3c7bbc8c22442d5b46722402a0a1bea P 8f9b0cac475346b7a96b276da650a22c reported cstate change: term changed from 0 to 1, leader changed from <none> to 8f9b0cac475346b7a96b276da650a22c (127.14.103.1). New cstate: current_term: 1 leader_uuid: "8f9b0cac475346b7a96b276da650a22c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9b0cac475346b7a96b276da650a22c" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 35705 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2bf93dc7aa034d5b8cab5efdc5dc0137" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 33765 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e3424a0a1ba94a0aa60efc89766f5e8e" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 37431 } health_report { overall_health: UNKNOWN } } }
I20260812 02:31:47.424892 16521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:31:47.420019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47858 (local address 127.14.103.1:35705)
0812 02:31:47.420189 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0812 02:31:47.420194 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:31:47.420212 (+ 18us) server_negotiation.cc:407] Connection header received
0812 02:31:47.420491 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:31:47.420494 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:31:47.420554 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:31:47.420668 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 02:31:47.421033 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:31:47.421565 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:31:47.422752 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:31:47.423068 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:31:47.423203 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:31:47.423646 (+ 443us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 02:31:47.424392 (+ 746us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 02:31:47.424551 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0812 02:31:47.424641 (+ 90us) server_negotiation.cc:299] Negotiation successful
0812 02:31:47.424714 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260812 02:31:47.433027 16297 raft_consensus.cc:1261] T b3c7bbc8c22442d5b46722402a0a1bea P 2bf93dc7aa034d5b8cab5efdc5dc0137 [term 1 FOLLOWER]: Refusing update from remote peer 8f9b0cac475346b7a96b276da650a22c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:31:47.434257 16449 raft_consensus.cc:1261] T b3c7bbc8c22442d5b46722402a0a1bea P e3424a0a1ba94a0aa60efc89766f5e8e [term 1 FOLLOWER]: Refusing update from remote peer 8f9b0cac475346b7a96b276da650a22c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:31:47.434888 16535 consensus_queue.cc:1035] T b3c7bbc8c22442d5b46722402a0a1bea P 8f9b0cac475346b7a96b276da650a22c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2bf93dc7aa034d5b8cab5efdc5dc0137" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 33765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:31:47.435232 16539 consensus_queue.cc:1035] T b3c7bbc8c22442d5b46722402a0a1bea P 8f9b0cac475346b7a96b276da650a22c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3424a0a1ba94a0aa60efc89766f5e8e" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 37431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:31:47.446535 16547 mvcc.cc:205] Tried to move back new op lower bound from 7317511812832346112 to 7317511812516790272. Current Snapshot: MvccSnapshot[applied={T|T < 7317511812832346112}]
I20260812 02:31:47.461635 15976 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:32902:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 02:31:47.461982 15976 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:32902:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 02:31:47.465291 15976 catalog_manager.cc:5782] T 00000000000000000000000000000000 P be19d8ff7b3e4e17a7341b403e1245df: Sending DeleteTablet for 3 replicas of tablet b3c7bbc8c22442d5b46722402a0a1bea
I20260812 02:31:47.466454 16121 tablet_service.cc:1514] Processing DeleteTablet for tablet b3c7bbc8c22442d5b46722402a0a1bea with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:31:47 UTC) from {username='kudu', principal='kudu/127.14.103.62@KRBTEST.COM'} at 127.0.0.1:47842
I20260812 02:31:47.466439 16274 tablet_service.cc:1514] Processing DeleteTablet for tablet b3c7bbc8c22442d5b46722402a0a1bea with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:31:47 UTC) from {username='kudu', principal='kudu/127.14.103.62@KRBTEST.COM'} at 127.0.0.1:51176
I20260812 02:31:47.467108 16428 tablet_service.cc:1514] Processing DeleteTablet for tablet b3c7bbc8c22442d5b46722402a0a1bea with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:31:47 UTC) from {username='kudu', principal='kudu/127.14.103.62@KRBTEST.COM'} at 127.0.0.1:48742
I20260812 02:31:47.467479 16555 tablet_replica.cc:330] T b3c7bbc8c22442d5b46722402a0a1bea P 8f9b0cac475346b7a96b276da650a22c: stopping tablet replica
I20260812 02:31:47.467619 16554 tablet_replica.cc:330] T b3c7bbc8c22442d5b46722402a0a1bea P 2bf93dc7aa034d5b8cab5efdc5dc0137: stopping tablet replica
I20260812 02:31:47.467706 15976 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:32902:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260812 02:31:47.467958 16554 raft_consensus.cc:2229] T b3c7bbc8c22442d5b46722402a0a1bea P 2bf93dc7aa034d5b8cab5efdc5dc0137 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 02:31:47.468158 16555 raft_consensus.cc:2229] T b3c7bbc8c22442d5b46722402a0a1bea P 8f9b0cac475346b7a96b276da650a22c [term 1 LEADER]: Raft consensus shutting down.
I20260812 02:31:47.468319 16554 raft_consensus.cc:2258] T b3c7bbc8c22442d5b46722402a0a1bea P 2bf93dc7aa034d5b8cab5efdc5dc0137 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260812 02:31:47.468310 15976 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 02:31:47.468797 16555 raft_consensus.cc:2258] T b3c7bbc8c22442d5b46722402a0a1bea P 8f9b0cac475346b7a96b276da650a22c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:31:47.469240 16556 tablet_replica.cc:330] T b3c7bbc8c22442d5b46722402a0a1bea P e3424a0a1ba94a0aa60efc89766f5e8e: stopping tablet replica
I20260812 02:31:47.469569 16556 raft_consensus.cc:2229] T b3c7bbc8c22442d5b46722402a0a1bea P e3424a0a1ba94a0aa60efc89766f5e8e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 02:31:47.469862 16556 raft_consensus.cc:2258] T b3c7bbc8c22442d5b46722402a0a1bea P e3424a0a1ba94a0aa60efc89766f5e8e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:31:47.469978 16554 ts_tablet_manager.cc:1905] T b3c7bbc8c22442d5b46722402a0a1bea P 2bf93dc7aa034d5b8cab5efdc5dc0137: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:31:47.471634 16555 ts_tablet_manager.cc:1905] T b3c7bbc8c22442d5b46722402a0a1bea P 8f9b0cac475346b7a96b276da650a22c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:31:47.473378 16554 ts_tablet_manager.cc:1918] T b3c7bbc8c22442d5b46722402a0a1bea P 2bf93dc7aa034d5b8cab5efdc5dc0137: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 02:31:47.473497 16554 log.cc:1196] T b3c7bbc8c22442d5b46722402a0a1bea P 2bf93dc7aa034d5b8cab5efdc5dc0137: Deleting WAL directory at /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501898713699-14748-0/minicluster-data/ts-1/wal/wals/b3c7bbc8c22442d5b46722402a0a1bea
I20260812 02:31:47.473860 16554 ts_tablet_manager.cc:1939] T b3c7bbc8c22442d5b46722402a0a1bea P 2bf93dc7aa034d5b8cab5efdc5dc0137: Deleting consensus metadata
I20260812 02:31:47.474884 16121 tablet_service.cc:1467] Processing CreateTablet for tablet 0db8ae38aa844fa2badfd4f2a734c4a0 (DEFAULT_TABLE table=test-table [id=8fbcff7a59bc4ffeba4c7a5d4eec2366]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:31:47.475126 16428 tablet_service.cc:1467] Processing CreateTablet for tablet 0db8ae38aa844fa2badfd4f2a734c4a0 (DEFAULT_TABLE table=test-table [id=8fbcff7a59bc4ffeba4c7a5d4eec2366]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:31:47.475232 16121 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0db8ae38aa844fa2badfd4f2a734c4a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:31:47.475387 16428 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0db8ae38aa844fa2badfd4f2a734c4a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:31:47.476812 16556 ts_tablet_manager.cc:1905] T b3c7bbc8c22442d5b46722402a0a1bea P e3424a0a1ba94a0aa60efc89766f5e8e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:31:47.477151 16555 ts_tablet_manager.cc:1918] T b3c7bbc8c22442d5b46722402a0a1bea P 8f9b0cac475346b7a96b276da650a22c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 02:31:47.477259 16555 log.cc:1196] T b3c7bbc8c22442d5b46722402a0a1bea P 8f9b0cac475346b7a96b276da650a22c: Deleting WAL directory at /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501898713699-14748-0/minicluster-data/ts-0/wal/wals/b3c7bbc8c22442d5b46722402a0a1bea
I20260812 02:31:47.477622 16555 ts_tablet_manager.cc:1939] T b3c7bbc8c22442d5b46722402a0a1bea P 8f9b0cac475346b7a96b276da650a22c: Deleting consensus metadata
I20260812 02:31:47.479151 15959 catalog_manager.cc:4841] TS 8f9b0cac475346b7a96b276da650a22c (127.14.103.1:35705): tablet b3c7bbc8c22442d5b46722402a0a1bea (table test-table [id=8683a37b764b46cfbf5650c10c16e0fc]) successfully deleted
I20260812 02:31:47.479218 16274 tablet_service.cc:1467] Processing CreateTablet for tablet 0db8ae38aa844fa2badfd4f2a734c4a0 (DEFAULT_TABLE table=test-table [id=8fbcff7a59bc4ffeba4c7a5d4eec2366]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:31:47.479519 16274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0db8ae38aa844fa2badfd4f2a734c4a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:31:47.479687 16556 ts_tablet_manager.cc:1918] T b3c7bbc8c22442d5b46722402a0a1bea P e3424a0a1ba94a0aa60efc89766f5e8e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 02:31:47.479868 16556 log.cc:1196] T b3c7bbc8c22442d5b46722402a0a1bea P e3424a0a1ba94a0aa60efc89766f5e8e: Deleting WAL directory at /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501898713699-14748-0/minicluster-data/ts-2/wal/wals/b3c7bbc8c22442d5b46722402a0a1bea
I20260812 02:31:47.479861 16531 tablet_bootstrap.cc:492] T 0db8ae38aa844fa2badfd4f2a734c4a0 P 8f9b0cac475346b7a96b276da650a22c: Bootstrap starting.
I20260812 02:31:47.481405 16556 ts_tablet_manager.cc:1939] T b3c7bbc8c22442d5b46722402a0a1bea P e3424a0a1ba94a0aa60efc89766f5e8e: Deleting consensus metadata
I20260812 02:31:47.481705 16531 tablet_bootstrap.cc:654] T 0db8ae38aa844fa2badfd4f2a734c4a0 P 8f9b0cac475346b7a96b276da650a22c: Neither blocks nor log segments found. Creating new log.
I20260812 02:31:47.483047 15960 catalog_manager.cc:4841] TS e3424a0a1ba94a0aa60efc89766f5e8e (127.14.103.3:37431): tablet b3c7bbc8c22442d5b46722402a0a1bea (table test-table [id=8683a37b764b46cfbf5650c10c16e0fc]) successfully deleted
I20260812 02:31:47.483556 16528 tablet_bootstrap.cc:492] T 0db8ae38aa844fa2badfd4f2a734c4a0 P 2bf93dc7aa034d5b8cab5efdc5dc0137: Bootstrap starting.
I20260812 02:31:47.483563 15962 catalog_manager.cc:4841] TS 2bf93dc7aa034d5b8cab5efdc5dc0137 (127.14.103.2:33765): tablet b3c7bbc8c22442d5b46722402a0a1bea (table test-table [id=8683a37b764b46cfbf5650c10c16e0fc]) successfully deleted
I20260812 02:31:47.483915 16527 tablet_bootstrap.cc:492] T 0db8ae38aa844fa2badfd4f2a734c4a0 P e3424a0a1ba94a0aa60efc89766f5e8e: Bootstrap starting.
I20260812 02:31:47.484726 16528 tablet_bootstrap.cc:654] T 0db8ae38aa844fa2badfd4f2a734c4a0 P 2bf93dc7aa034d5b8cab5efdc5dc0137: Neither blocks nor log segments found. Creating new log.
I20260812 02:31:47.484876 16527 tablet_bootstrap.cc:654] T 0db8ae38aa844fa2badfd4f2a734c4a0 P e3424a0a1ba94a0aa60efc89766f5e8e: Neither blocks nor log segments found. Creating new log.
I20260812 02:31:47.486140 16531 tablet_bootstrap.cc:492] T 0db8ae38aa844fa2badfd4f2a734c4a0 P 8f9b0cac475346b7a96b276da650a22c: No bootstrap required, opened a new log
I20260812 02:31:47.486224 16527 tablet_bootstrap.cc:492] T 0db8ae38aa844fa2badfd4f2a734c4a0 P e3424a0a1ba94a0aa60efc89766f5e8e: No bootstrap required, opened a new log
I20260812 02:31:47.486250 16531 ts_tablet_manager.cc:1397] T 0db8ae38aa844fa2badfd4f2a734c4a0 P 8f9b0cac475346b7a96b276da650a22c: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.001s
I20260812 02:31:47.486317 16527 ts_tablet_manager.cc:1397] T 0db8ae38aa844fa2badfd4f2a734c4a0 P e3424a0a1ba94a0aa60efc89766f5e8e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260812 02:31:47.486805 16527 raft_consensus.cc:348] T 0db8ae38aa844fa2badfd4f2a734c4a0 P e3424a0a1ba94a0aa60efc89766f5e8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9b0cac475346b7a96b276da650a22c" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 35705 } } peers { permanent_uuid: "2bf93dc7aa034d5b8cab5efdc5dc0137" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 33765 } } peers { permanent_uuid: "e3424a0a1ba94a0aa60efc89766f5e8e" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 37431 } }
I20260812 02:31:47.486833 16531 raft_consensus.cc:348] T 0db8ae38aa844fa2badfd4f2a734c4a0 P 8f9b0cac475346b7a96b276da650a22c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9b0cac475346b7a96b276da650a22c" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 35705 } } peers { permanent_uuid: "2bf93dc7aa034d5b8cab5efdc5dc0137" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 33765 } } peers { permanent_uuid: "e3424a0a1ba94a0aa60efc89766f5e8e" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 37431 } }
I20260812 02:31:47.486943 16527 raft_consensus.cc:374] T 0db8ae38aa844fa2badfd4f2a734c4a0 P e3424a0a1ba94a0aa60efc89766f5e8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:31:47.486951 16531 raft_consensus.cc:374] T 0db8ae38aa844fa2badfd4f2a734c4a0 P 8f9b0cac475346b7a96b276da650a22c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:31:47.487012 16527 raft_consensus.cc:729] T 0db8ae38aa844fa2badfd4f2a734c4a0 P e3424a0a1ba94a0aa60efc89766f5e8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3424a0a1ba94a0aa60efc89766f5e8e, State: Initialized, Role: FOLLOWER
I20260812 02:31:47.487030 16531 raft_consensus.cc:729] T 0db8ae38aa844fa2badfd4f2a734c4a0 P 8f9b0cac475346b7a96b276da650a22c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f9b0cac475346b7a96b276da650a22c, State: Initialized, Role: FOLLOWER
I20260812 02:31:47.487146 16527 consensus_queue.cc:260] T 0db8ae38aa844fa2badfd4f2a734c4a0 P e3424a0a1ba94a0aa60efc89766f5e8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9b0cac475346b7a96b276da650a22c" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 35705 } } peers { permanent_uuid: "2bf93dc7aa034d5b8cab5efdc5dc0137" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 33765 } } peers { permanent_uuid: "e3424a0a1ba94a0aa60efc89766f5e8e" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 37431 } }
I20260812 02:31:47.487160 16531 consensus_queue.cc:260] T 0db8ae38aa844fa2badfd4f2a734c4a0 P 8f9b0cac475346b7a96b276da650a22c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9b0cac475346b7a96b276da650a22c" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 35705 } } peers { permanent_uuid: "2bf93dc7aa034d5b8cab5efdc5dc0137" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 33765 } } peers { permanent_uuid: "e3424a0a1ba94a0aa60efc89766f5e8e" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 37431 } }
I20260812 02:31:47.487490 16531 ts_tablet_manager.cc:1428] T 0db8ae38aa844fa2badfd4f2a734c4a0 P 8f9b0cac475346b7a96b276da650a22c: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260812 02:31:47.488636 16528 tablet_bootstrap.cc:492] T 0db8ae38aa844fa2badfd4f2a734c4a0 P 2bf93dc7aa034d5b8cab5efdc5dc0137: No bootstrap required, opened a new log
I20260812 02:31:47.488754 16528 ts_tablet_manager.cc:1397] T 0db8ae38aa844fa2badfd4f2a734c4a0 P 2bf93dc7aa034d5b8cab5efdc5dc0137: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260812 02:31:47.488997 16527 ts_tablet_manager.cc:1428] T 0db8ae38aa844fa2badfd4f2a734c4a0 P e3424a0a1ba94a0aa60efc89766f5e8e: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20260812 02:31:47.489275 16528 raft_consensus.cc:348] T 0db8ae38aa844fa2badfd4f2a734c4a0 P 2bf93dc7aa034d5b8cab5efdc5dc0137 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9b0cac475346b7a96b276da650a22c" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 35705 } } peers { permanent_uuid: "2bf93dc7aa034d5b8cab5efdc5dc0137" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 33765 } } peers { permanent_uuid: "e3424a0a1ba94a0aa60efc89766f5e8e" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 37431 } }
I20260812 02:31:47.489403 16528 raft_consensus.cc:374] T 0db8ae38aa844fa2badfd4f2a734c4a0 P 2bf93dc7aa034d5b8cab5efdc5dc0137 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:31:47.489440 16528 raft_consensus.cc:729] T 0db8ae38aa844fa2badfd4f2a734c4a0 P 2bf93dc7aa034d5b8cab5efdc5dc0137 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2bf93dc7aa034d5b8cab5efdc5dc0137, State: Initialized, Role: FOLLOWER
I20260812 02:31:47.489580 16528 consensus_queue.cc:260] T 0db8ae38aa844fa2badfd4f2a734c4a0 P 2bf93dc7aa034d5b8cab5efdc5dc0137 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9b0cac475346b7a96b276da650a22c" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 35705 } } peers { permanent_uuid: "2bf93dc7aa034d5b8cab5efdc5dc0137" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 33765 } } peers { permanent_uuid: "e3424a0a1ba94a0aa60efc89766f5e8e" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 37431 } }
I20260812 02:31:47.489879 16528 ts_tablet_manager.cc:1428] T 0db8ae38aa844fa2badfd4f2a734c4a0 P 2bf93dc7aa034d5b8cab5efdc5dc0137: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260812 02:31:47.492287 16539 raft_consensus.cc:482] T 0db8ae38aa844fa2badfd4f2a734c4a0 P 8f9b0cac475346b7a96b276da650a22c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 02:31:47.492441 16539 raft_consensus.cc:504] T 0db8ae38aa844fa2badfd4f2a734c4a0 P 8f9b0cac475346b7a96b276da650a22c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9b0cac475346b7a96b276da650a22c" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 35705 } } peers { permanent_uuid: "2bf93dc7aa034d5b8cab5efdc5dc0137" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 33765 } } peers { permanent_uuid: "e3424a0a1ba94a0aa60efc89766f5e8e" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 37431 } }
I20260812 02:31:47.492911 16539 leader_election.cc:288] T 0db8ae38aa844fa2badfd4f2a734c4a0 P 8f9b0cac475346b7a96b276da650a22c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2bf93dc7aa034d5b8cab5efdc5dc0137 (127.14.103.2:33765), e3424a0a1ba94a0aa60efc89766f5e8e (127.14.103.3:37431)
I20260812 02:31:47.493192 16297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0db8ae38aa844fa2badfd4f2a734c4a0" candidate_uuid: "8f9b0cac475346b7a96b276da650a22c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2bf93dc7aa034d5b8cab5efdc5dc0137" is_pre_election: true
I20260812 02:31:47.493350 16297 raft_consensus.cc:2454] T 0db8ae38aa844fa2badfd4f2a734c4a0 P 2bf93dc7aa034d5b8cab5efdc5dc0137 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f9b0cac475346b7a96b276da650a22c in term 0.
I20260812 02:31:47.493557 16449 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0db8ae38aa844fa2badfd4f2a734c4a0" candidate_uuid: "8f9b0cac475346b7a96b276da650a22c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3424a0a1ba94a0aa60efc89766f5e8e" is_pre_election: true
I20260812 02:31:47.493678 16074 leader_election.cc:302] T 0db8ae38aa844fa2badfd4f2a734c4a0 P 8f9b0cac475346b7a96b276da650a22c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2bf93dc7aa034d5b8cab5efdc5dc0137, 8f9b0cac475346b7a96b276da650a22c; no voters:
I20260812 02:31:47.493691 16449 raft_consensus.cc:2454] T 0db8ae38aa844fa2badfd4f2a734c4a0 P e3424a0a1ba94a0aa60efc89766f5e8e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f9b0cac475346b7a96b276da650a22c in term 0.
I20260812 02:31:47.493841 16539 raft_consensus.cc:2781] T 0db8ae38aa844fa2badfd4f2a734c4a0 P 8f9b0cac475346b7a96b276da650a22c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 02:31:47.493896 16539 raft_consensus.cc:482] T 0db8ae38aa844fa2badfd4f2a734c4a0 P 8f9b0cac475346b7a96b276da650a22c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 02:31:47.493934 16539 raft_consensus.cc:3037] T 0db8ae38aa844fa2badfd4f2a734c4a0 P 8f9b0cac475346b7a96b276da650a22c [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:31:47.495323 16539 raft_consensus.cc:504] T 0db8ae38aa844fa2badfd4f2a734c4a0 P 8f9b0cac475346b7a96b276da650a22c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9b0cac475346b7a96b276da650a22c" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 35705 } } peers { permanent_uuid: "2bf93dc7aa034d5b8cab5efdc5dc0137" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 33765 } } peers { permanent_uuid: "e3424a0a1ba94a0aa60efc89766f5e8e" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 37431 } }
I20260812 02:31:47.495723 16539 leader_election.cc:288] T 0db8ae38aa844fa2badfd4f2a734c4a0 P 8f9b0cac475346b7a96b276da650a22c [CANDIDATE]: Term 1 election: Requested vote from peers 2bf93dc7aa034d5b8cab5efdc5dc0137 (127.14.103.2:33765), e3424a0a1ba94a0aa60efc89766f5e8e (127.14.103.3:37431)
I20260812 02:31:47.496096 16449 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0db8ae38aa844fa2badfd4f2a734c4a0" candidate_uuid: "8f9b0cac475346b7a96b276da650a22c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3424a0a1ba94a0aa60efc89766f5e8e"
I20260812 02:31:47.496238 16449 raft_consensus.cc:3037] T 0db8ae38aa844fa2badfd4f2a734c4a0 P e3424a0a1ba94a0aa60efc89766f5e8e [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:31:47.497534 16449 raft_consensus.cc:2454] T 0db8ae38aa844fa2badfd4f2a734c4a0 P e3424a0a1ba94a0aa60efc89766f5e8e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f9b0cac475346b7a96b276da650a22c in term 1.
I20260812 02:31:47.497877 16297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0db8ae38aa844fa2badfd4f2a734c4a0" candidate_uuid: "8f9b0cac475346b7a96b276da650a22c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2bf93dc7aa034d5b8cab5efdc5dc0137"
I20260812 02:31:47.498027 16297 raft_consensus.cc:3037] T 0db8ae38aa844fa2badfd4f2a734c4a0 P 2bf93dc7aa034d5b8cab5efdc5dc0137 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:31:47.499410 16072 leader_election.cc:302] T 0db8ae38aa844fa2badfd4f2a734c4a0 P 8f9b0cac475346b7a96b276da650a22c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f9b0cac475346b7a96b276da650a22c, e3424a0a1ba94a0aa60efc89766f5e8e; no voters:
I20260812 02:31:47.499681 16539 raft_consensus.cc:2781] T 0db8ae38aa844fa2badfd4f2a734c4a0 P 8f9b0cac475346b7a96b276da650a22c [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:31:47.499692 16297 raft_consensus.cc:2454] T 0db8ae38aa844fa2badfd4f2a734c4a0 P 2bf93dc7aa034d5b8cab5efdc5dc0137 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f9b0cac475346b7a96b276da650a22c in term 1.
I20260812 02:31:47.499861 16539 raft_consensus.cc:686] T 0db8ae38aa844fa2badfd4f2a734c4a0 P 8f9b0cac475346b7a96b276da650a22c [term 1 LEADER]: Becoming Leader. State: Replica: 8f9b0cac475346b7a96b276da650a22c, State: Running, Role: LEADER
I20260812 02:31:47.500046 16539 consensus_queue.cc:237] T 0db8ae38aa844fa2badfd4f2a734c4a0 P 8f9b0cac475346b7a96b276da650a22c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9b0cac475346b7a96b276da650a22c" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 35705 } } peers { permanent_uuid: "2bf93dc7aa034d5b8cab5efdc5dc0137" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 33765 } } peers { permanent_uuid: "e3424a0a1ba94a0aa60efc89766f5e8e" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 37431 } }
I20260812 02:31:47.502293 15975 catalog_manager.cc:5495] T 0db8ae38aa844fa2badfd4f2a734c4a0 P 8f9b0cac475346b7a96b276da650a22c reported cstate change: term changed from 0 to 1, leader changed from <none> to 8f9b0cac475346b7a96b276da650a22c (127.14.103.1). New cstate: current_term: 1 leader_uuid: "8f9b0cac475346b7a96b276da650a22c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9b0cac475346b7a96b276da650a22c" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 35705 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2bf93dc7aa034d5b8cab5efdc5dc0137" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 33765 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e3424a0a1ba94a0aa60efc89766f5e8e" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 37431 } health_report { overall_health: UNKNOWN } } }
Aug 12 02:31:47 dist-test-slave-2w3w krb5kdc[15928](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501905, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.62@KRBTEST.COM for kudu/127.14.103.62@KRBTEST.COM
I20260812 02:31:47.556039 16569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:31:47.545551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32918 (local address 127.14.103.62:40425)
0812 02:31:47.545790 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0812 02:31:47.545795 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:31:47.545830 (+ 35us) server_negotiation.cc:407] Connection header received
0812 02:31:47.546001 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:31:47.546004 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:31:47.546056 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:31:47.546186 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:31:47.546932 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:31:47.547809 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:31:47.548711 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:31:47.549018 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:31:47.549255 (+ 237us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:31:47.551911 (+ 2656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:31:47.551915 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:31:47.551918 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:31:47.551946 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:31:47.553789 (+ 1843us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:31:47.554401 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:31:47.554404 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:31:47.554405 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:31:47.554493 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:31:47.554801 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:31:47.554804 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:31:47.554805 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:31:47.555105 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:31:47.555219 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0812 02:31:47.555626 (+ 407us) server_negotiation.cc:299] Negotiation successful
0812 02:31:47.555862 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":152,"server-negotiator.queue_time_us":171,"thread_start_us":73,"threads_started":1}
I20260812 02:31:47.556048 16568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:31:47.545443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:40425 (local address 127.0.0.1:32918)
0812 02:31:47.545727 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0812 02:31:47.545749 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0812 02:31:47.545884 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:31:47.546232 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:31:47.546235 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:31:47.546691 (+ 456us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:31:47.546802 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:31:47.546808 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:31:47.547990 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:31:47.547992 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:31:47.548572 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:31:47.548580 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:31:47.549178 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:31:47.549181 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:31:47.549295 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:31:47.549958 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:31:47.549971 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:31:47.551752 (+ 1781us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:31:47.553893 (+ 2141us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:31:47.553912 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:31:47.553920 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:31:47.554275 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:31:47.554591 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:31:47.554594 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:31:47.554596 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:31:47.554694 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:31:47.555299 (+ 605us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:31:47.555307 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:31:47.555447 (+ 140us) client_negotiation.cc:769] Sending connection context
0812 02:31:47.555579 (+ 132us) client_negotiation.cc:240] Negotiation successful
0812 02:31:47.555786 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"mutex_wait_us":3,"thread_start_us":99,"threads_started":1}
I20260812 02:31:47.561836 15975 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.14.103.62@KRBTEST.COM'} at 127.0.0.1:32918:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260812 02:31:47.562790 15975 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 02:31:47.569087 16274 tablet_service.cc:1467] Processing CreateTablet for tablet f673a73348e147f1978fdbc96dac7321 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=209358b0bb9d43ef98fabf41bcea6443]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260812 02:31:47.569080 16121 tablet_service.cc:1467] Processing CreateTablet for tablet f673a73348e147f1978fdbc96dac7321 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=209358b0bb9d43ef98fabf41bcea6443]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260812 02:31:47.569559 16274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f673a73348e147f1978fdbc96dac7321. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:31:47.569561 16121 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f673a73348e147f1978fdbc96dac7321. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:31:47.572630 16528 tablet_bootstrap.cc:492] T f673a73348e147f1978fdbc96dac7321 P 2bf93dc7aa034d5b8cab5efdc5dc0137: Bootstrap starting.
I20260812 02:31:47.572630 16531 tablet_bootstrap.cc:492] T f673a73348e147f1978fdbc96dac7321 P 8f9b0cac475346b7a96b276da650a22c: Bootstrap starting.
I20260812 02:31:47.573210 16428 tablet_service.cc:1467] Processing CreateTablet for tablet f673a73348e147f1978fdbc96dac7321 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=209358b0bb9d43ef98fabf41bcea6443]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260812 02:31:47.573593 16428 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f673a73348e147f1978fdbc96dac7321. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:31:47.573652 16528 tablet_bootstrap.cc:654] T f673a73348e147f1978fdbc96dac7321 P 2bf93dc7aa034d5b8cab5efdc5dc0137: Neither blocks nor log segments found. Creating new log.
I20260812 02:31:47.573673 16531 tablet_bootstrap.cc:654] T f673a73348e147f1978fdbc96dac7321 P 8f9b0cac475346b7a96b276da650a22c: Neither blocks nor log segments found. Creating new log.
I20260812 02:31:47.575467 16528 tablet_bootstrap.cc:492] T f673a73348e147f1978fdbc96dac7321 P 2bf93dc7aa034d5b8cab5efdc5dc0137: No bootstrap required, opened a new log
I20260812 02:31:47.575459 16531 tablet_bootstrap.cc:492] T f673a73348e147f1978fdbc96dac7321 P 8f9b0cac475346b7a96b276da650a22c: No bootstrap required, opened a new log
I20260812 02:31:47.575568 16531 ts_tablet_manager.cc:1397] T f673a73348e147f1978fdbc96dac7321 P 8f9b0cac475346b7a96b276da650a22c: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260812 02:31:47.575568 16528 ts_tablet_manager.cc:1397] T f673a73348e147f1978fdbc96dac7321 P 2bf93dc7aa034d5b8cab5efdc5dc0137: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260812 02:31:47.576278 16531 raft_consensus.cc:348] T f673a73348e147f1978fdbc96dac7321 P 8f9b0cac475346b7a96b276da650a22c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9b0cac475346b7a96b276da650a22c" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 35705 } } peers { permanent_uuid: "2bf93dc7aa034d5b8cab5efdc5dc0137" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 33765 } } peers { permanent_uuid: "e3424a0a1ba94a0aa60efc89766f5e8e" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 37431 } }
I20260812 02:31:47.576458 16531 raft_consensus.cc:374] T f673a73348e147f1978fdbc96dac7321 P 8f9b0cac475346b7a96b276da650a22c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:31:47.576500 16531 raft_consensus.cc:729] T f673a73348e147f1978fdbc96dac7321 P 8f9b0cac475346b7a96b276da650a22c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f9b0cac475346b7a96b276da650a22c, State: Initialized, Role: FOLLOWER
I20260812 02:31:47.576555 16528 raft_consensus.cc:348] T f673a73348e147f1978fdbc96dac7321 P 2bf93dc7aa034d5b8cab5efdc5dc0137 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9b0cac475346b7a96b276da650a22c" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 35705 } } peers { permanent_uuid: "2bf93dc7aa034d5b8cab5efdc5dc0137" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 33765 } } peers { permanent_uuid: "e3424a0a1ba94a0aa60efc89766f5e8e" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 37431 } }
I20260812 02:31:47.576679 16528 raft_consensus.cc:374] T f673a73348e147f1978fdbc96dac7321 P 2bf93dc7aa034d5b8cab5efdc5dc0137 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:31:47.576717 16528 raft_consensus.cc:729] T f673a73348e147f1978fdbc96dac7321 P 2bf93dc7aa034d5b8cab5efdc5dc0137 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2bf93dc7aa034d5b8cab5efdc5dc0137, State: Initialized, Role: FOLLOWER
I20260812 02:31:47.576661 16531 consensus_queue.cc:260] T f673a73348e147f1978fdbc96dac7321 P 8f9b0cac475346b7a96b276da650a22c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9b0cac475346b7a96b276da650a22c" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 35705 } } peers { permanent_uuid: "2bf93dc7aa034d5b8cab5efdc5dc0137" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 33765 } } peers { permanent_uuid: "e3424a0a1ba94a0aa60efc89766f5e8e" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 37431 } }
I20260812 02:31:47.576845 16528 consensus_queue.cc:260] T f673a73348e147f1978fdbc96dac7321 P 2bf93dc7aa034d5b8cab5efdc5dc0137 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9b0cac475346b7a96b276da650a22c" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 35705 } } peers { permanent_uuid: "2bf93dc7aa034d5b8cab5efdc5dc0137" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 33765 } } peers { permanent_uuid: "e3424a0a1ba94a0aa60efc89766f5e8e" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 37431 } }
I20260812 02:31:47.577008 16531 ts_tablet_manager.cc:1428] T f673a73348e147f1978fdbc96dac7321 P 8f9b0cac475346b7a96b276da650a22c: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260812 02:31:47.577266 16528 ts_tablet_manager.cc:1428] T f673a73348e147f1978fdbc96dac7321 P 2bf93dc7aa034d5b8cab5efdc5dc0137: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260812 02:31:47.577418 16539 tablet_replica.cc:439] T f673a73348e147f1978fdbc96dac7321 P 8f9b0cac475346b7a96b276da650a22c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9b0cac475346b7a96b276da650a22c" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 35705 } } peers { permanent_uuid: "2bf93dc7aa034d5b8cab5efdc5dc0137" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 33765 } } peers { permanent_uuid: "e3424a0a1ba94a0aa60efc89766f5e8e" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 37431 } } }
I20260812 02:31:47.577381 16534 tablet_replica.cc:439] T f673a73348e147f1978fdbc96dac7321 P 2bf93dc7aa034d5b8cab5efdc5dc0137: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9b0cac475346b7a96b276da650a22c" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 35705 } } peers { permanent_uuid: "2bf93dc7aa034d5b8cab5efdc5dc0137" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 33765 } } peers { permanent_uuid: "e3424a0a1ba94a0aa60efc89766f5e8e" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 37431 } } }
I20260812 02:31:47.577565 16539 tablet_replica.cc:442] T f673a73348e147f1978fdbc96dac7321 P 8f9b0cac475346b7a96b276da650a22c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 02:31:47.577577 16534 tablet_replica.cc:442] T f673a73348e147f1978fdbc96dac7321 P 2bf93dc7aa034d5b8cab5efdc5dc0137: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 02:31:47.579926 16527 tablet_bootstrap.cc:492] T f673a73348e147f1978fdbc96dac7321 P e3424a0a1ba94a0aa60efc89766f5e8e: Bootstrap starting.
I20260812 02:31:47.581426 16527 tablet_bootstrap.cc:654] T f673a73348e147f1978fdbc96dac7321 P e3424a0a1ba94a0aa60efc89766f5e8e: Neither blocks nor log segments found. Creating new log.
I20260812 02:31:47.583257 16527 tablet_bootstrap.cc:492] T f673a73348e147f1978fdbc96dac7321 P e3424a0a1ba94a0aa60efc89766f5e8e: No bootstrap required, opened a new log
I20260812 02:31:47.583520 16527 ts_tablet_manager.cc:1397] T f673a73348e147f1978fdbc96dac7321 P e3424a0a1ba94a0aa60efc89766f5e8e: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260812 02:31:47.584017 16527 raft_consensus.cc:348] T f673a73348e147f1978fdbc96dac7321 P e3424a0a1ba94a0aa60efc89766f5e8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9b0cac475346b7a96b276da650a22c" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 35705 } } peers { permanent_uuid: "2bf93dc7aa034d5b8cab5efdc5dc0137" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 33765 } } peers { permanent_uuid: "e3424a0a1ba94a0aa60efc89766f5e8e" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 37431 } }
I20260812 02:31:47.584168 16527 raft_consensus.cc:374] T f673a73348e147f1978fdbc96dac7321 P e3424a0a1ba94a0aa60efc89766f5e8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:31:47.584203 16527 raft_consensus.cc:729] T f673a73348e147f1978fdbc96dac7321 P e3424a0a1ba94a0aa60efc89766f5e8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3424a0a1ba94a0aa60efc89766f5e8e, State: Initialized, Role: FOLLOWER
I20260812 02:31:47.584373 16527 consensus_queue.cc:260] T f673a73348e147f1978fdbc96dac7321 P e3424a0a1ba94a0aa60efc89766f5e8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9b0cac475346b7a96b276da650a22c" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 35705 } } peers { permanent_uuid: "2bf93dc7aa034d5b8cab5efdc5dc0137" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 33765 } } peers { permanent_uuid: "e3424a0a1ba94a0aa60efc89766f5e8e" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 37431 } }
I20260812 02:31:47.584684 16527 ts_tablet_manager.cc:1428] T f673a73348e147f1978fdbc96dac7321 P e3424a0a1ba94a0aa60efc89766f5e8e: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20260812 02:31:47.584698 16532 tablet_replica.cc:439] T f673a73348e147f1978fdbc96dac7321 P e3424a0a1ba94a0aa60efc89766f5e8e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9b0cac475346b7a96b276da650a22c" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 35705 } } peers { permanent_uuid: "2bf93dc7aa034d5b8cab5efdc5dc0137" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 33765 } } peers { permanent_uuid: "e3424a0a1ba94a0aa60efc89766f5e8e" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 37431 } } }
I20260812 02:31:47.584910 16532 tablet_replica.cc:442] T f673a73348e147f1978fdbc96dac7321 P e3424a0a1ba94a0aa60efc89766f5e8e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 02:31:47.904313 16539 consensus_queue.cc:1035] T 0db8ae38aa844fa2badfd4f2a734c4a0 P 8f9b0cac475346b7a96b276da650a22c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3424a0a1ba94a0aa60efc89766f5e8e" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 37431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:31:47.906445 16539 raft_consensus.cc:482] T f673a73348e147f1978fdbc96dac7321 P 8f9b0cac475346b7a96b276da650a22c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 02:31:47.906584 16539 raft_consensus.cc:504] T f673a73348e147f1978fdbc96dac7321 P 8f9b0cac475346b7a96b276da650a22c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9b0cac475346b7a96b276da650a22c" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 35705 } } peers { permanent_uuid: "2bf93dc7aa034d5b8cab5efdc5dc0137" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 33765 } } peers { permanent_uuid: "e3424a0a1ba94a0aa60efc89766f5e8e" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 37431 } }
I20260812 02:31:47.907050 16539 leader_election.cc:288] T f673a73348e147f1978fdbc96dac7321 P 8f9b0cac475346b7a96b276da650a22c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2bf93dc7aa034d5b8cab5efdc5dc0137 (127.14.103.2:33765), e3424a0a1ba94a0aa60efc89766f5e8e (127.14.103.3:37431)
I20260812 02:31:47.907469 16297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f673a73348e147f1978fdbc96dac7321" candidate_uuid: "8f9b0cac475346b7a96b276da650a22c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2bf93dc7aa034d5b8cab5efdc5dc0137" is_pre_election: true
I20260812 02:31:47.907465 16449 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f673a73348e147f1978fdbc96dac7321" candidate_uuid: "8f9b0cac475346b7a96b276da650a22c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3424a0a1ba94a0aa60efc89766f5e8e" is_pre_election: true
I20260812 02:31:47.907615 16297 raft_consensus.cc:2454] T f673a73348e147f1978fdbc96dac7321 P 2bf93dc7aa034d5b8cab5efdc5dc0137 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f9b0cac475346b7a96b276da650a22c in term 0.
I20260812 02:31:47.907615 16449 raft_consensus.cc:2454] T f673a73348e147f1978fdbc96dac7321 P e3424a0a1ba94a0aa60efc89766f5e8e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f9b0cac475346b7a96b276da650a22c in term 0.
I20260812 02:31:47.908074 16074 leader_election.cc:302] T f673a73348e147f1978fdbc96dac7321 P 8f9b0cac475346b7a96b276da650a22c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2bf93dc7aa034d5b8cab5efdc5dc0137, 8f9b0cac475346b7a96b276da650a22c; no voters:
I20260812 02:31:47.908252 16535 raft_consensus.cc:2781] T f673a73348e147f1978fdbc96dac7321 P 8f9b0cac475346b7a96b276da650a22c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 02:31:47.908303 16535 raft_consensus.cc:482] T f673a73348e147f1978fdbc96dac7321 P 8f9b0cac475346b7a96b276da650a22c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 02:31:47.908329 16535 raft_consensus.cc:3037] T f673a73348e147f1978fdbc96dac7321 P 8f9b0cac475346b7a96b276da650a22c [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:31:47.908675 16539 consensus_queue.cc:1035] T 0db8ae38aa844fa2badfd4f2a734c4a0 P 8f9b0cac475346b7a96b276da650a22c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2bf93dc7aa034d5b8cab5efdc5dc0137" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 33765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:31:47.909688 16535 raft_consensus.cc:504] T f673a73348e147f1978fdbc96dac7321 P 8f9b0cac475346b7a96b276da650a22c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9b0cac475346b7a96b276da650a22c" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 35705 } } peers { permanent_uuid: "2bf93dc7aa034d5b8cab5efdc5dc0137" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 33765 } } peers { permanent_uuid: "e3424a0a1ba94a0aa60efc89766f5e8e" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 37431 } }
I20260812 02:31:47.910082 16535 leader_election.cc:288] T f673a73348e147f1978fdbc96dac7321 P 8f9b0cac475346b7a96b276da650a22c [CANDIDATE]: Term 1 election: Requested vote from peers 2bf93dc7aa034d5b8cab5efdc5dc0137 (127.14.103.2:33765), e3424a0a1ba94a0aa60efc89766f5e8e (127.14.103.3:37431)
I20260812 02:31:47.910604 16296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f673a73348e147f1978fdbc96dac7321" candidate_uuid: "8f9b0cac475346b7a96b276da650a22c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2bf93dc7aa034d5b8cab5efdc5dc0137"
I20260812 02:31:47.910624 16447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f673a73348e147f1978fdbc96dac7321" candidate_uuid: "8f9b0cac475346b7a96b276da650a22c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3424a0a1ba94a0aa60efc89766f5e8e"
I20260812 02:31:47.910737 16447 raft_consensus.cc:3037] T f673a73348e147f1978fdbc96dac7321 P e3424a0a1ba94a0aa60efc89766f5e8e [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:31:47.910737 16296 raft_consensus.cc:3037] T f673a73348e147f1978fdbc96dac7321 P 2bf93dc7aa034d5b8cab5efdc5dc0137 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:31:47.911948 16447 raft_consensus.cc:2454] T f673a73348e147f1978fdbc96dac7321 P e3424a0a1ba94a0aa60efc89766f5e8e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f9b0cac475346b7a96b276da650a22c in term 1.
I20260812 02:31:47.911944 16296 raft_consensus.cc:2454] T f673a73348e147f1978fdbc96dac7321 P 2bf93dc7aa034d5b8cab5efdc5dc0137 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f9b0cac475346b7a96b276da650a22c in term 1.
I20260812 02:31:47.912317 16072 leader_election.cc:302] T f673a73348e147f1978fdbc96dac7321 P 8f9b0cac475346b7a96b276da650a22c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f9b0cac475346b7a96b276da650a22c, e3424a0a1ba94a0aa60efc89766f5e8e; no voters:
I20260812 02:31:47.912496 16535 raft_consensus.cc:2781] T f673a73348e147f1978fdbc96dac7321 P 8f9b0cac475346b7a96b276da650a22c [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:31:47.912650 16535 raft_consensus.cc:686] T f673a73348e147f1978fdbc96dac7321 P 8f9b0cac475346b7a96b276da650a22c [term 1 LEADER]: Becoming Leader. State: Replica: 8f9b0cac475346b7a96b276da650a22c, State: Running, Role: LEADER
I20260812 02:31:47.912817 16535 consensus_queue.cc:237] T f673a73348e147f1978fdbc96dac7321 P 8f9b0cac475346b7a96b276da650a22c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9b0cac475346b7a96b276da650a22c" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 35705 } } peers { permanent_uuid: "2bf93dc7aa034d5b8cab5efdc5dc0137" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 33765 } } peers { permanent_uuid: "e3424a0a1ba94a0aa60efc89766f5e8e" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 37431 } }
I20260812 02:31:47.913522 16575 tablet_replica.cc:439] T f673a73348e147f1978fdbc96dac7321 P 8f9b0cac475346b7a96b276da650a22c: TxnStatusTablet state changed. Reason: New leader 8f9b0cac475346b7a96b276da650a22c. Latest consensus state: current_term: 1 leader_uuid: "8f9b0cac475346b7a96b276da650a22c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9b0cac475346b7a96b276da650a22c" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 35705 } } peers { permanent_uuid: "2bf93dc7aa034d5b8cab5efdc5dc0137" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 33765 } } peers { permanent_uuid: "e3424a0a1ba94a0aa60efc89766f5e8e" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 37431 } } }
I20260812 02:31:47.913642 16575 tablet_replica.cc:442] T f673a73348e147f1978fdbc96dac7321 P 8f9b0cac475346b7a96b276da650a22c: This TxnStatusTablet replica's current role is: LEADER
I20260812 02:31:47.914314 16578 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260812 02:31:47.914521 16578 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260812 02:31:47.914551 15974 catalog_manager.cc:5495] T f673a73348e147f1978fdbc96dac7321 P 8f9b0cac475346b7a96b276da650a22c reported cstate change: term changed from 0 to 1, leader changed from <none> to 8f9b0cac475346b7a96b276da650a22c (127.14.103.1). New cstate: current_term: 1 leader_uuid: "8f9b0cac475346b7a96b276da650a22c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9b0cac475346b7a96b276da650a22c" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 35705 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2bf93dc7aa034d5b8cab5efdc5dc0137" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 33765 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e3424a0a1ba94a0aa60efc89766f5e8e" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 37431 } health_report { overall_health: UNKNOWN } } }
I20260812 02:31:47.951200 16581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:31:47.947761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47862 (local address 127.14.103.1:35705)
0812 02:31:47.948046 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0812 02:31:47.948056 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 02:31:47.948068 (+ 12us) server_negotiation.cc:407] Connection header received
0812 02:31:47.948113 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:31:47.948115 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:31:47.948159 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:31:47.948253 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 02:31:47.948516 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:31:47.949050 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:31:47.949981 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:31:47.950228 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:31:47.950342 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:31:47.950656 (+ 314us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 02:31:47.950742 (+ 86us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 02:31:47.950822 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0812 02:31:47.950949 (+ 127us) server_negotiation.cc:299] Negotiation successful
0812 02:31:47.951036 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":199,"thread_start_us":81,"threads_started":1}
I20260812 02:31:47.951195 16568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:31:47.947599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.1:35705 (local address 127.0.0.1:47862)
0812 02:31:47.947765 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0812 02:31:47.947783 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 02:31:47.947915 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:31:47.948276 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:31:47.948278 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:31:47.948286 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 02:31:47.948402 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:31:47.948407 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:31:47.949163 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:31:47.949167 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:31:47.949844 (+ 677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:31:47.949854 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:31:47.950349 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:31:47.950353 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:31:47.950451 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:31:47.950502 (+ 51us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 02:31:47.950827 (+ 325us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 02:31:47.950847 (+ 20us) client_negotiation.cc:769] Sending connection context
0812 02:31:47.950932 (+ 85us) client_negotiation.cc:240] Negotiation successful
0812 02:31:47.951057 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260812 02:31:47.954365 16296 raft_consensus.cc:1261] T f673a73348e147f1978fdbc96dac7321 P 2bf93dc7aa034d5b8cab5efdc5dc0137 [term 1 FOLLOWER]: Refusing update from remote peer 8f9b0cac475346b7a96b276da650a22c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:31:47.954885 16575 consensus_queue.cc:1035] T f673a73348e147f1978fdbc96dac7321 P 8f9b0cac475346b7a96b276da650a22c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2bf93dc7aa034d5b8cab5efdc5dc0137" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 33765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:31:47.956691 16534 tablet_replica.cc:439] T f673a73348e147f1978fdbc96dac7321 P 2bf93dc7aa034d5b8cab5efdc5dc0137: TxnStatusTablet state changed. Reason: New leader 8f9b0cac475346b7a96b276da650a22c. Latest consensus state: current_term: 1 leader_uuid: "8f9b0cac475346b7a96b276da650a22c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9b0cac475346b7a96b276da650a22c" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 35705 } } peers { permanent_uuid: "2bf93dc7aa034d5b8cab5efdc5dc0137" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 33765 } } peers { permanent_uuid: "e3424a0a1ba94a0aa60efc89766f5e8e" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 37431 } } }
I20260812 02:31:47.956816 16534 tablet_replica.cc:442] T f673a73348e147f1978fdbc96dac7321 P 2bf93dc7aa034d5b8cab5efdc5dc0137: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 02:31:47.958250 16575 tablet_replica.cc:439] T f673a73348e147f1978fdbc96dac7321 P 8f9b0cac475346b7a96b276da650a22c: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8f9b0cac475346b7a96b276da650a22c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9b0cac475346b7a96b276da650a22c" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 35705 } } peers { permanent_uuid: "2bf93dc7aa034d5b8cab5efdc5dc0137" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 33765 } } peers { permanent_uuid: "e3424a0a1ba94a0aa60efc89766f5e8e" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 37431 } } }
I20260812 02:31:47.958388 16575 tablet_replica.cc:442] T f673a73348e147f1978fdbc96dac7321 P 8f9b0cac475346b7a96b276da650a22c: This TxnStatusTablet replica's current role is: LEADER
I20260812 02:31:47.959340 16544 mvcc.cc:205] Tried to move back new op lower bound from 7317511814976032768 to 7317511814812344320. Current Snapshot: MvccSnapshot[applied={T|T < 7317511814976032768}]
I20260812 02:31:47.959419 16447 raft_consensus.cc:1261] T f673a73348e147f1978fdbc96dac7321 P e3424a0a1ba94a0aa60efc89766f5e8e [term 1 FOLLOWER]: Refusing update from remote peer 8f9b0cac475346b7a96b276da650a22c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:31:47.960134 16535 consensus_queue.cc:1035] T f673a73348e147f1978fdbc96dac7321 P 8f9b0cac475346b7a96b276da650a22c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3424a0a1ba94a0aa60efc89766f5e8e" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 37431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:31:47.960543 16534 tablet_replica.cc:439] T f673a73348e147f1978fdbc96dac7321 P 2bf93dc7aa034d5b8cab5efdc5dc0137: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8f9b0cac475346b7a96b276da650a22c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9b0cac475346b7a96b276da650a22c" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 35705 } } peers { permanent_uuid: "2bf93dc7aa034d5b8cab5efdc5dc0137" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 33765 } } peers { permanent_uuid: "e3424a0a1ba94a0aa60efc89766f5e8e" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 37431 } } }
I20260812 02:31:47.960675 16534 tablet_replica.cc:442] T f673a73348e147f1978fdbc96dac7321 P 2bf93dc7aa034d5b8cab5efdc5dc0137: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 02:31:47.961829 16586 tablet_replica.cc:439] T f673a73348e147f1978fdbc96dac7321 P e3424a0a1ba94a0aa60efc89766f5e8e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8f9b0cac475346b7a96b276da650a22c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9b0cac475346b7a96b276da650a22c" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 35705 } } peers { permanent_uuid: "2bf93dc7aa034d5b8cab5efdc5dc0137" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 33765 } } peers { permanent_uuid: "e3424a0a1ba94a0aa60efc89766f5e8e" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 37431 } } }
I20260812 02:31:47.961977 16586 tablet_replica.cc:442] T f673a73348e147f1978fdbc96dac7321 P e3424a0a1ba94a0aa60efc89766f5e8e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 02:31:47.962659 16532 tablet_replica.cc:439] T f673a73348e147f1978fdbc96dac7321 P e3424a0a1ba94a0aa60efc89766f5e8e: TxnStatusTablet state changed. Reason: New leader 8f9b0cac475346b7a96b276da650a22c. Latest consensus state: current_term: 1 leader_uuid: "8f9b0cac475346b7a96b276da650a22c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9b0cac475346b7a96b276da650a22c" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 35705 } } peers { permanent_uuid: "2bf93dc7aa034d5b8cab5efdc5dc0137" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 33765 } } peers { permanent_uuid: "e3424a0a1ba94a0aa60efc89766f5e8e" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 37431 } } }
I20260812 02:31:47.962775 16532 tablet_replica.cc:442] T f673a73348e147f1978fdbc96dac7321 P e3424a0a1ba94a0aa60efc89766f5e8e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 02:31:47.991688 16590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:31:47.987224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.1:35705 (local address 127.14.103.1:46791)
0812 02:31:47.987581 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0812 02:31:47.987595 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 02:31:47.987710 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:31:47.987982 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:31:47.987985 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:31:47.987992 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 02:31:47.988090 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:31:47.988095 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:31:47.989256 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:31:47.989259 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:31:47.989915 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:31:47.989925 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:31:47.990785 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:31:47.990788 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:31:47.990869 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:31:47.990928 (+ 59us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 02:31:47.991384 (+ 456us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 02:31:47.991393 (+ 9us) client_negotiation.cc:769] Sending connection context
0812 02:31:47.991457 (+ 64us) client_negotiation.cc:240] Negotiation successful
0812 02:31:47.991543 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":143,"threads_started":1}
I20260812 02:31:47.991704 16581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:31:47.987325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.1:46791 (local address 127.14.103.1:35705)
0812 02:31:47.987483 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0812 02:31:47.987488 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:31:47.987671 (+ 183us) server_negotiation.cc:407] Connection header received
0812 02:31:47.987803 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:31:47.987807 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:31:47.987866 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:31:47.987972 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 02:31:47.988206 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:31:47.989089 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:31:47.990058 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:31:47.990527 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:31:47.990656 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:31:47.991163 (+ 507us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 02:31:47.991261 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 02:31:47.991340 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0812 02:31:47.991493 (+ 153us) server_negotiation.cc:299] Negotiation successful
0812 02:31:47.991555 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":56}
I20260812 02:31:47.994102 16575 tablet_replica.cc:439] T f673a73348e147f1978fdbc96dac7321 P 8f9b0cac475346b7a96b276da650a22c: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8f9b0cac475346b7a96b276da650a22c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f9b0cac475346b7a96b276da650a22c" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 35705 } } peers { permanent_uuid: "2bf93dc7aa034d5b8cab5efdc5dc0137" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 33765 } } peers { permanent_uuid: "e3424a0a1ba94a0aa60efc89766f5e8e" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 37431 } } }
I20260812 02:31:47.994263 16575 tablet_replica.cc:442] T f673a73348e147f1978fdbc96dac7321 P 8f9b0cac475346b7a96b276da650a22c: This TxnStatusTablet replica's current role is: LEADER
I20260812 02:31:48.044203 15974 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:32902:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 02:31:48.044477 15974 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:32902:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 02:31:48.047540 15974 catalog_manager.cc:5782] T 00000000000000000000000000000000 P be19d8ff7b3e4e17a7341b403e1245df: Sending DeleteTablet for 3 replicas of tablet 0db8ae38aa844fa2badfd4f2a734c4a0
I20260812 02:31:48.048352 16274 tablet_service.cc:1514] Processing DeleteTablet for tablet 0db8ae38aa844fa2badfd4f2a734c4a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:31:48 UTC) from {username='kudu', principal='kudu/127.14.103.62@KRBTEST.COM'} at 127.0.0.1:51176
I20260812 02:31:48.048381 16428 tablet_service.cc:1514] Processing DeleteTablet for tablet 0db8ae38aa844fa2badfd4f2a734c4a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:31:48 UTC) from {username='kudu', principal='kudu/127.14.103.62@KRBTEST.COM'} at 127.0.0.1:48742
I20260812 02:31:48.048426 16119 tablet_service.cc:1514] Processing DeleteTablet for tablet 0db8ae38aa844fa2badfd4f2a734c4a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:31:48 UTC) from {username='kudu', principal='kudu/127.14.103.62@KRBTEST.COM'} at 127.0.0.1:47842
I20260812 02:31:48.048698 16594 tablet_replica.cc:330] T 0db8ae38aa844fa2badfd4f2a734c4a0 P 2bf93dc7aa034d5b8cab5efdc5dc0137: stopping tablet replica
I20260812 02:31:48.048698 16595 tablet_replica.cc:330] T 0db8ae38aa844fa2badfd4f2a734c4a0 P e3424a0a1ba94a0aa60efc89766f5e8e: stopping tablet replica
I20260812 02:31:48.048820 16594 raft_consensus.cc:2229] T 0db8ae38aa844fa2badfd4f2a734c4a0 P 2bf93dc7aa034d5b8cab5efdc5dc0137 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 02:31:48.048854 16595 raft_consensus.cc:2229] T 0db8ae38aa844fa2badfd4f2a734c4a0 P e3424a0a1ba94a0aa60efc89766f5e8e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 02:31:48.048962 16595 raft_consensus.cc:2258] T 0db8ae38aa844fa2badfd4f2a734c4a0 P e3424a0a1ba94a0aa60efc89766f5e8e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:31:48.048964 16594 raft_consensus.cc:2258] T 0db8ae38aa844fa2badfd4f2a734c4a0 P 2bf93dc7aa034d5b8cab5efdc5dc0137 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:31:48.049778 16595 ts_tablet_manager.cc:1905] T 0db8ae38aa844fa2badfd4f2a734c4a0 P e3424a0a1ba94a0aa60efc89766f5e8e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:31:48.049903 16596 tablet_replica.cc:330] T 0db8ae38aa844fa2badfd4f2a734c4a0 P 8f9b0cac475346b7a96b276da650a22c: stopping tablet replica
I20260812 02:31:48.050099 16596 raft_consensus.cc:2229] T 0db8ae38aa844fa2badfd4f2a734c4a0 P 8f9b0cac475346b7a96b276da650a22c [term 1 LEADER]: Raft consensus shutting down.
I20260812 02:31:48.050372 16596 raft_consensus.cc:2258] T 0db8ae38aa844fa2badfd4f2a734c4a0 P 8f9b0cac475346b7a96b276da650a22c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:31:48.050946 16594 ts_tablet_manager.cc:1905] T 0db8ae38aa844fa2badfd4f2a734c4a0 P 2bf93dc7aa034d5b8cab5efdc5dc0137: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:31:48.051241 16596 ts_tablet_manager.cc:1905] T 0db8ae38aa844fa2badfd4f2a734c4a0 P 8f9b0cac475346b7a96b276da650a22c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:31:48.052939 16595 ts_tablet_manager.cc:1918] T 0db8ae38aa844fa2badfd4f2a734c4a0 P e3424a0a1ba94a0aa60efc89766f5e8e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 02:31:48.053031 16595 log.cc:1196] T 0db8ae38aa844fa2badfd4f2a734c4a0 P e3424a0a1ba94a0aa60efc89766f5e8e: Deleting WAL directory at /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501898713699-14748-0/minicluster-data/ts-2/wal/wals/0db8ae38aa844fa2badfd4f2a734c4a0
I20260812 02:31:48.053318 16595 ts_tablet_manager.cc:1939] T 0db8ae38aa844fa2badfd4f2a734c4a0 P e3424a0a1ba94a0aa60efc89766f5e8e: Deleting consensus metadata
I20260812 02:31:48.054342 16594 ts_tablet_manager.cc:1918] T 0db8ae38aa844fa2badfd4f2a734c4a0 P 2bf93dc7aa034d5b8cab5efdc5dc0137: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 02:31:48.054421 16594 log.cc:1196] T 0db8ae38aa844fa2badfd4f2a734c4a0 P 2bf93dc7aa034d5b8cab5efdc5dc0137: Deleting WAL directory at /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501898713699-14748-0/minicluster-data/ts-1/wal/wals/0db8ae38aa844fa2badfd4f2a734c4a0
I20260812 02:31:48.054675 16594 ts_tablet_manager.cc:1939] T 0db8ae38aa844fa2badfd4f2a734c4a0 P 2bf93dc7aa034d5b8cab5efdc5dc0137: Deleting consensus metadata
I20260812 02:31:48.055691 15962 catalog_manager.cc:4841] TS 2bf93dc7aa034d5b8cab5efdc5dc0137 (127.14.103.2:33765): tablet 0db8ae38aa844fa2badfd4f2a734c4a0 (table test-table [id=8fbcff7a59bc4ffeba4c7a5d4eec2366]) successfully deleted
I20260812 02:31:48.059944 15960 catalog_manager.cc:4841] TS e3424a0a1ba94a0aa60efc89766f5e8e (127.14.103.3:37431): tablet 0db8ae38aa844fa2badfd4f2a734c4a0 (table test-table [id=8fbcff7a59bc4ffeba4c7a5d4eec2366]) successfully deleted
I20260812 02:31:48.062204 16596 ts_tablet_manager.cc:1918] T 0db8ae38aa844fa2badfd4f2a734c4a0 P 8f9b0cac475346b7a96b276da650a22c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 02:31:48.062342 16596 log.cc:1196] T 0db8ae38aa844fa2badfd4f2a734c4a0 P 8f9b0cac475346b7a96b276da650a22c: Deleting WAL directory at /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786501898713699-14748-0/minicluster-data/ts-0/wal/wals/0db8ae38aa844fa2badfd4f2a734c4a0
I20260812 02:31:48.062644 16596 ts_tablet_manager.cc:1939] T 0db8ae38aa844fa2badfd4f2a734c4a0 P 8f9b0cac475346b7a96b276da650a22c: Deleting consensus metadata
I20260812 02:31:48.063674 15959 catalog_manager.cc:4841] TS 8f9b0cac475346b7a96b276da650a22c (127.14.103.1:35705): tablet 0db8ae38aa844fa2badfd4f2a734c4a0 (table test-table [id=8fbcff7a59bc4ffeba4c7a5d4eec2366]) successfully deleted
Aug 12 02:31:48 dist-test-slave-2w3w krb5kdc[15928](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501906, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.14.103.1@KRBTEST.COM
I20260812 02:31:48.069872 16581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:31:48.055132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47866 (local address 127.14.103.1:35705)
0812 02:31:48.055344 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0812 02:31:48.055349 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:31:48.055367 (+ 18us) server_negotiation.cc:407] Connection header received
0812 02:31:48.055416 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:31:48.055419 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:31:48.055475 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:31:48.055580 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:31:48.056463 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:31:48.057047 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:31:48.057896 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:31:48.058129 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:31:48.058238 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:31:48.065461 (+ 7223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:31:48.065475 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:31:48.065491 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:31:48.065523 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:31:48.067796 (+ 2273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:31:48.068344 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:31:48.068351 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:31:48.068353 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:31:48.068465 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:31:48.068780 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:31:48.068784 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:31:48.068786 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:31:48.069075 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:31:48.069319 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0812 02:31:48.069500 (+ 181us) server_negotiation.cc:299] Negotiation successful
0812 02:31:48.069640 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129}
W20260812 02:31:48.070461 16187 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47866
I20260812 02:31:48.082646 16600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:31:48.072385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32930 (local address 127.14.103.62:40425)
0812 02:31:48.072691 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0812 02:31:48.072695 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:31:48.073027 (+ 332us) server_negotiation.cc:407] Connection header received
0812 02:31:48.073120 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:31:48.073124 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:31:48.073184 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:31:48.073301 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:31:48.074074 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:31:48.075012 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:31:48.075856 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:31:48.076251 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:31:48.076365 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:31:48.078290 (+ 1925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:31:48.078294 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:31:48.078297 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:31:48.078331 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:31:48.080311 (+ 1980us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:31:48.080946 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:31:48.080949 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:31:48.080952 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:31:48.081055 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:31:48.081484 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:31:48.081487 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:31:48.081489 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:31:48.081744 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:31:48.081914 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0812 02:31:48.082259 (+ 345us) server_negotiation.cc:299] Negotiation successful
0812 02:31:48.082429 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":224,"thread_start_us":95,"threads_started":1}
W20260812 02:31:48.083171 15974 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:32930
I20260812 02:31:48.085170 14748 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbep4M4/build/debug/bin/kudu with pid 16054
I20260812 02:31:48.100778 14748 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbep4M4/build/debug/bin/kudu with pid 16202
I20260812 02:31:48.113034 14748 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbep4M4/build/debug/bin/kudu with pid 16358
I20260812 02:31:48.123001 14748 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbep4M4/build/debug/bin/kudu with pid 15942
2026-08-12T02:31:48Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (3578 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501898713699-14748-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 02:31:48 dist-test-slave-2w3w krb5kdc[16605](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 02:31:48 dist-test-slave-2w3w krb5kdc[16605](info): set up 2 sockets
Aug 12 02:31:48 dist-test-slave-2w3w krb5kdc[16605](info): commencing operation
krb5kdc: starting...
W20260812 02:31:49.333585 14748 mini_kdc.cc:121] Time spent starting KDC: real 1.184s user 0.001s sys 0.009s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501898713699-14748-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501898713699-14748-0/krb5kdc/test-user.keytab.
Aug 12 02:31:49 dist-test-slave-2w3w krb5kdc[16605](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501909, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T02:31:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T02:31:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.103.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.103.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.103.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.62.keytab.
Entry for principal kudu/127.14.103.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.62.keytab.
I20260812 02:31:49.501052 14748 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbep4M4/build/debug/bin/kudu
/tmp/dist-test-taskbep4M4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501898713699-14748-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501898713699-14748-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501898713699-14748-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501898713699-14748-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.103.62:43757
--webserver_interface=127.14.103.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.62.keytab
--principal=kudu/127.14.103.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.103.20:37045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.103.62:43757
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501898713699-14748-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501898713699-14748-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501898713699-14748-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:31:49.642663 16626 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:31:49.643078 16626 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:31:49.643168 16626 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:31:49.648494 16626 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:31:49.648649 16626 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260812 02:31:49.648690 16626 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:31:49.648746 16626 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:31:49.648788 16626 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:31:49.648833 16626 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 02:31:49.655500 16626 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.103.20:37045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501898713699-14748-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501898713699-14748-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.103.62:43757
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.62.keytab
--principal=kudu/127.14.103.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.103.62:43757
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501898713699-14748-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.103.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16626
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501898713699-14748-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:31:49.656932 16626 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:31:49.658174 16626 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:31:49.670230 16634 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:31:49.670361 16637 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:31:49.670390 16633 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:31:49.671077 16626 server_base.cc:1034] running on GCE node
I20260812 02:31:49.671835 16626 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:31:49.673112 16626 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:31:49.674396 16626 hybrid_clock.cc:648] HybridClock initialized: now 1786501909674377 us; error 43 us; skew 500 ppm
Aug 12 02:31:49 dist-test-slave-2w3w krb5kdc[16605](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501909, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:31:49.678047 16626 init.cc:377] Logged in from keytab as kudu/127.14.103.62@KRBTEST.COM (short username kudu)
I20260812 02:31:49.679407 16626 webserver.cc:460] Webserver started at http://127.14.103.62:41547/ using document root <none> and password file <none>
I20260812 02:31:49.680166 16626 fs_manager.cc:359] Metadata directory not provided
I20260812 02:31:49.680276 16626 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:31:49.680552 16626 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:31:49.683382 16626 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501898713699-14748-0/minicluster-data/master-0/data/instance:
uuid: "43b9d054e04b419f9b79d092ff15ba6b"
format_stamp: "Formatted at 2026-08-12 02:31:49 on dist-test-slave-2w3w"
server_key: "194324af145173db40e78266e3883927"
server_key_iv: "5e4ab6d0369aff5fee05c5a8403d2e77"
server_key_version: "encryptionkey@0"
I20260812 02:31:49.684127 16626 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501898713699-14748-0/minicluster-data/master-0/wal/instance:
uuid: "43b9d054e04b419f9b79d092ff15ba6b"
format_stamp: "Formatted at 2026-08-12 02:31:49 on dist-test-slave-2w3w"
server_key: "194324af145173db40e78266e3883927"
server_key_iv: "5e4ab6d0369aff5fee05c5a8403d2e77"
server_key_version: "encryptionkey@0"
I20260812 02:31:49.688686 16626 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260812 02:31:49.692058 16644 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:31:49.693704 16626 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260812 02:31:49.693945 16626 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501898713699-14748-0/minicluster-data/master-0/data,/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501898713699-14748-0/minicluster-data/master-0/wal
uuid: "43b9d054e04b419f9b79d092ff15ba6b"
format_stamp: "Formatted at 2026-08-12 02:31:49 on dist-test-slave-2w3w"
server_key: "194324af145173db40e78266e3883927"
server_key_iv: "5e4ab6d0369aff5fee05c5a8403d2e77"
server_key_version: "encryptionkey@0"
I20260812 02:31:49.694118 16626 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501898713699-14748-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501898713699-14748-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501898713699-14748-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:31:49.727739 16626 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:31:49.731820 16626 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:31:49.732069 16626 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:31:49.742527 16713 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.103.62:43757 every 8 connection(s)
I20260812 02:31:49.742527 16626 rpc_server.cc:307] RPC server started. Bound to: 127.14.103.62:43757
I20260812 02:31:49.743950 16626 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501898713699-14748-0/minicluster-data/master-0/data/info.pb
I20260812 02:31:49.747467 16714 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:31:49.753561 14748 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbep4M4/build/debug/bin/kudu as pid 16626
I20260812 02:31:49.753701 14748 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501898713699-14748-0/minicluster-data/master-0/wal/instance
I20260812 02:31:49.753983 14748 external_mini_cluster.cc:1428] Setting key 33690e853e7b59f16acda84cc9a2130d
I20260812 02:31:49.754417 16714 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43b9d054e04b419f9b79d092ff15ba6b: Bootstrap starting.
I20260812 02:31:49.757272 16714 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 43b9d054e04b419f9b79d092ff15ba6b: Neither blocks nor log segments found. Creating new log.
I20260812 02:31:49.758131 16714 log.cc:824] T 00000000000000000000000000000000 P 43b9d054e04b419f9b79d092ff15ba6b: Log is configured to *not* fsync() on all Append() calls
I20260812 02:31:49.760556 16714 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43b9d054e04b419f9b79d092ff15ba6b: No bootstrap required, opened a new log
Aug 12 02:31:49 dist-test-slave-2w3w krb5kdc[16605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501909, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.103.62@KRBTEST.COM
I20260812 02:31:49.765277 16714 raft_consensus.cc:348] T 00000000000000000000000000000000 P 43b9d054e04b419f9b79d092ff15ba6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43b9d054e04b419f9b79d092ff15ba6b" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 43757 } }
I20260812 02:31:49.765748 16714 raft_consensus.cc:374] T 00000000000000000000000000000000 P 43b9d054e04b419f9b79d092ff15ba6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:31:49.765838 16714 raft_consensus.cc:729] T 00000000000000000000000000000000 P 43b9d054e04b419f9b79d092ff15ba6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43b9d054e04b419f9b79d092ff15ba6b, State: Initialized, Role: FOLLOWER
I20260812 02:31:49.766424 16714 consensus_queue.cc:260] T 00000000000000000000000000000000 P 43b9d054e04b419f9b79d092ff15ba6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43b9d054e04b419f9b79d092ff15ba6b" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 43757 } }
I20260812 02:31:49.766587 16714 raft_consensus.cc:388] T 00000000000000000000000000000000 P 43b9d054e04b419f9b79d092ff15ba6b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 02:31:49.766685 16714 raft_consensus.cc:482] T 00000000000000000000000000000000 P 43b9d054e04b419f9b79d092ff15ba6b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 02:31:49.766835 16714 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 43b9d054e04b419f9b79d092ff15ba6b [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:31:49.767978 16714 raft_consensus.cc:504] T 00000000000000000000000000000000 P 43b9d054e04b419f9b79d092ff15ba6b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43b9d054e04b419f9b79d092ff15ba6b" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 43757 } }
I20260812 02:31:49.768404 16714 leader_election.cc:302] T 00000000000000000000000000000000 P 43b9d054e04b419f9b79d092ff15ba6b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 43b9d054e04b419f9b79d092ff15ba6b; no voters:
I20260812 02:31:49.768783 16714 leader_election.cc:288] T 00000000000000000000000000000000 P 43b9d054e04b419f9b79d092ff15ba6b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 02:31:49.768962 16720 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 43b9d054e04b419f9b79d092ff15ba6b [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:31:49.769235 16720 raft_consensus.cc:686] T 00000000000000000000000000000000 P 43b9d054e04b419f9b79d092ff15ba6b [term 1 LEADER]: Becoming Leader. State: Replica: 43b9d054e04b419f9b79d092ff15ba6b, State: Running, Role: LEADER
I20260812 02:31:49.769614 16720 consensus_queue.cc:237] T 00000000000000000000000000000000 P 43b9d054e04b419f9b79d092ff15ba6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43b9d054e04b419f9b79d092ff15ba6b" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 43757 } }
I20260812 02:31:49.770303 16714 sys_catalog.cc:564] T 00000000000000000000000000000000 P 43b9d054e04b419f9b79d092ff15ba6b [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:31:49.771483 16722 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43b9d054e04b419f9b79d092ff15ba6b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "43b9d054e04b419f9b79d092ff15ba6b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43b9d054e04b419f9b79d092ff15ba6b" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 43757 } } }
I20260812 02:31:49.771643 16722 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43b9d054e04b419f9b79d092ff15ba6b [sys.catalog]: This master's current role is: LEADER
I20260812 02:31:49.771981 16721 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43b9d054e04b419f9b79d092ff15ba6b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 43b9d054e04b419f9b79d092ff15ba6b. Latest consensus state: current_term: 1 leader_uuid: "43b9d054e04b419f9b79d092ff15ba6b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43b9d054e04b419f9b79d092ff15ba6b" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 43757 } } }
I20260812 02:31:49.772064 16721 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43b9d054e04b419f9b79d092ff15ba6b [sys.catalog]: This master's current role is: LEADER
I20260812 02:31:49.772082 16729 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 02:31:49.773119 16718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:31:49.755619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51176 (local address 127.14.103.62:43757)
0812 02:31:49.756179 (+ 560us) server_negotiation.cc:206] Beginning negotiation
0812 02:31:49.756195 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0812 02:31:49.756234 (+ 39us) server_negotiation.cc:407] Connection header received
0812 02:31:49.756977 (+ 743us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:31:49.756998 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:31:49.757412 (+ 414us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:31:49.757811 (+ 399us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:31:49.758731 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:31:49.759880 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:31:49.761490 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:31:49.762647 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:31:49.762919 (+ 272us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:31:49.766303 (+ 3384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:31:49.766310 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:31:49.766326 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:31:49.766370 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:31:49.768475 (+ 2105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:31:49.769114 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:31:49.769122 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:31:49.769129 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:31:49.769341 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:31:49.769695 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:31:49.769699 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:31:49.769701 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:31:49.770269 (+ 568us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:31:49.770493 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0812 02:31:49.771249 (+ 756us) server_negotiation.cc:299] Negotiation successful
0812 02:31:49.771637 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":404,"thread_start_us":176,"threads_started":1}
I20260812 02:31:49.776448 16729 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 02:31:49.783478 16729 catalog_manager.cc:1347] Generated new cluster ID: 31c653dc8ba845d49fcc057868bb4a2c
I20260812 02:31:49.783598 16729 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 02:31:49.802127 16729 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 02:31:49.803341 16729 catalog_manager.cc:1504] Loading token signing keys...
I20260812 02:31:49.816632 16729 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 43b9d054e04b419f9b79d092ff15ba6b: Generated new TSK 0
I20260812 02:31:49.817708 16729 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.103.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.103.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.103.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.1.keytab.
Entry for principal kudu/127.14.103.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.1.keytab.
I20260812 02:31:49.855877 14748 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbep4M4/build/debug/bin/kudu
/tmp/dist-test-taskbep4M4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501898713699-14748-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501898713699-14748-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501898713699-14748-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501898713699-14748-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.103.1:0
--local_ip_for_outbound_sockets=127.14.103.1
--webserver_interface=127.14.103.1
--webserver_port=0
--tserver_master_addrs=127.14.103.62:43757
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.1.keytab
--principal=kudu/127.14.103.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.103.20:37045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501898713699-14748-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501898713699-14748-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501898713699-14748-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:31:49.997536 16742 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:31:49.997886 16742 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:31:49.997956 16742 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:31:50.002790 16742 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:31:50.002897 16742 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:31:50.002948 16742 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260812 02:31:50.003014 16742 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 02:31:50.003084 16742 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.103.1
I20260812 02:31:50.009546 16742 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.103.20:37045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501898713699-14748-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501898713699-14748-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.1.keytab
--principal=kudu/127.14.103.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.103.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501898713699-14748-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.103.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.14.103.62:43757
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16742
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.103.1
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501898713699-14748-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:31:50.011086 16742 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:31:50.012311 16742 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:31:50.024616 16749 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:31:50.024818 16742 server_base.cc:1034] running on GCE node
W20260812 02:31:50.024591 16748 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:31:50.025182 16751 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:31:50.026625 16742 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:31:50.027736 16742 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:31:50.029057 16742 hybrid_clock.cc:648] HybridClock initialized: now 1786501910029004 us; error 54 us; skew 500 ppm
Aug 12 02:31:50 dist-test-slave-2w3w krb5kdc[16605](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501910, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:31:50.032806 16742 init.cc:377] Logged in from keytab as kudu/127.14.103.1@KRBTEST.COM (short username kudu)
I20260812 02:31:50.033922 16742 webserver.cc:460] Webserver started at http://127.14.103.1:35297/ using document root <none> and password file <none>
I20260812 02:31:50.034693 16742 fs_manager.cc:359] Metadata directory not provided
I20260812 02:31:50.034883 16742 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:31:50.035508 16742 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:31:50.038590 16742 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501898713699-14748-0/minicluster-data/ts-0/data/instance:
uuid: "ac78ac69480a4042a6278f187e3b4d61"
format_stamp: "Formatted at 2026-08-12 02:31:50 on dist-test-slave-2w3w"
server_key: "ceeb4eb7c446034f0dad1481fac743c5"
server_key_iv: "f732188290926d08db2a8af9b4af45ff"
server_key_version: "encryptionkey@0"
I20260812 02:31:50.039304 16742 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501898713699-14748-0/minicluster-data/ts-0/wal/instance:
uuid: "ac78ac69480a4042a6278f187e3b4d61"
format_stamp: "Formatted at 2026-08-12 02:31:50 on dist-test-slave-2w3w"
server_key: "ceeb4eb7c446034f0dad1481fac743c5"
server_key_iv: "f732188290926d08db2a8af9b4af45ff"
server_key_version: "encryptionkey@0"
I20260812 02:31:50.043654 16742 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.003s
I20260812 02:31:50.046607 16759 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:31:50.048236 16742 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 02:31:50.048421 16742 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501898713699-14748-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501898713699-14748-0/minicluster-data/ts-0/wal
uuid: "ac78ac69480a4042a6278f187e3b4d61"
format_stamp: "Formatted at 2026-08-12 02:31:50 on dist-test-slave-2w3w"
server_key: "ceeb4eb7c446034f0dad1481fac743c5"
server_key_iv: "f732188290926d08db2a8af9b4af45ff"
server_key_version: "encryptionkey@0"
I20260812 02:31:50.048573 16742 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501898713699-14748-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501898713699-14748-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501898713699-14748-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:31:50.077705 16742 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:31:50.081686 16742 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:31:50.081943 16742 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:31:50.083724 16742 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:31:50.083802 16742 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:31:50.083869 16742 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:31:50.083916 16742 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 12 02:31:50 dist-test-slave-2w3w krb5kdc[16605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501910, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.1@KRBTEST.COM for kudu/127.14.103.62@KRBTEST.COM
I20260812 02:31:50.097252 16742 rpc_server.cc:307] RPC server started. Bound to: 127.14.103.1:38185
I20260812 02:31:50.097368 16884 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.103.1:38185 every 8 connection(s)
I20260812 02:31:50.098371 16742 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501898713699-14748-0/minicluster-data/ts-0/data/info.pb
I20260812 02:31:50.101504 16718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:31:50.084856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.1:49193 (local address 127.14.103.62:43757)
0812 02:31:50.085010 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0812 02:31:50.085014 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:31:50.086383 (+ 1369us) server_negotiation.cc:407] Connection header received
0812 02:31:50.087662 (+ 1279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:31:50.087665 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:31:50.087733 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:31:50.087838 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:31:50.089179 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:31:50.089790 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:31:50.091143 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:31:50.091526 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:31:50.091652 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:31:50.096108 (+ 4456us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:31:50.096113 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:31:50.096116 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:31:50.096159 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:31:50.098046 (+ 1887us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:31:50.099047 (+ 1001us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:31:50.099051 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:31:50.099053 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:31:50.099182 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:31:50.099804 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:31:50.099807 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:31:50.099809 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:31:50.100150 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:31:50.100419 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0812 02:31:50.101015 (+ 596us) server_negotiation.cc:299] Negotiation successful
0812 02:31:50.101217 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260812 02:31:50.102705 16769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:31:50.085366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:43757 (local address 127.14.103.1:49193)
0812 02:31:50.086179 (+ 813us) negotiation.cc:107] Waiting for socket to connect
0812 02:31:50.086229 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0812 02:31:50.087387 (+ 1158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:31:50.088075 (+ 688us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:31:50.088083 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:31:50.088475 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:31:50.088994 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:31:50.089008 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:31:50.089938 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:31:50.089941 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:31:50.090996 (+ 1055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:31:50.091010 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:31:50.091707 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:31:50.091711 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:31:50.091936 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:31:50.092697 (+ 761us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:31:50.092723 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:31:50.095599 (+ 2876us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:31:50.098325 (+ 2726us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:31:50.098335 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:31:50.098352 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:31:50.098783 (+ 431us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:31:50.099320 (+ 537us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:31:50.099324 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:31:50.099326 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:31:50.099652 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:31:50.100409 (+ 757us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:31:50.100416 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:31:50.100711 (+ 295us) client_negotiation.cc:769] Sending connection context
0812 02:31:50.100949 (+ 238us) client_negotiation.cc:240] Negotiation successful
0812 02:31:50.101420 (+ 471us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":581,"thread_start_us":222,"threads_started":1}
I20260812 02:31:50.106527 14748 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbep4M4/build/debug/bin/kudu as pid 16742
I20260812 02:31:50.106691 14748 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501898713699-14748-0/minicluster-data/ts-0/wal/instance
I20260812 02:31:50.106822 16885 heartbeater.cc:347] Connected to a master server at 127.14.103.62:43757
I20260812 02:31:50.107072 14748 external_mini_cluster.cc:1428] Setting key e4c1649dee6c296527873eabd0ed69ef
I20260812 02:31:50.107087 16885 heartbeater.cc:464] Registering TS with master...
I20260812 02:31:50.107870 16885 heartbeater.cc:511] Master 127.14.103.62:43757 requested a full tablet report, sending...
I20260812 02:31:50.110316 16664 ts_manager.cc:194] Registered new tserver with Master: ac78ac69480a4042a6278f187e3b4d61 (127.14.103.1:38185)
I20260812 02:31:50.112226 16664 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.103.1@KRBTEST.COM'} at 127.14.103.1:49193
I20260812 02:31:50.120951 16718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:31:50.110290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.1:35177 (local address 127.14.103.62:43757)
0812 02:31:50.110541 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0812 02:31:50.110546 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:31:50.110956 (+ 410us) server_negotiation.cc:407] Connection header received
0812 02:31:50.111142 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:31:50.111146 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:31:50.111202 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:31:50.111312 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:31:50.112038 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:31:50.112700 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:31:50.113787 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:31:50.114115 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:31:50.114243 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:31:50.116078 (+ 1835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:31:50.116081 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:31:50.116083 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:31:50.116111 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:31:50.117720 (+ 1609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:31:50.119058 (+ 1338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:31:50.119122 (+ 64us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:31:50.119123 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:31:50.119231 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:31:50.119665 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:31:50.119668 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:31:50.119669 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:31:50.119881 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:31:50.119967 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0812 02:31:50.120658 (+ 691us) server_negotiation.cc:299] Negotiation successful
0812 02:31:50.120789 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":133}
I20260812 02:31:50.121028 16892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:31:50.110065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:43757 (local address 127.14.103.1:35177)
0812 02:31:50.110817 (+ 752us) negotiation.cc:107] Waiting for socket to connect
0812 02:31:50.110838 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0812 02:31:50.111015 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:31:50.111354 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:31:50.111360 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:31:50.111674 (+ 314us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:31:50.111864 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:31:50.111906 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:31:50.112843 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:31:50.112847 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:31:50.113571 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:31:50.113587 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:31:50.114297 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:31:50.114303 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:31:50.114448 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:31:50.115249 (+ 801us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:31:50.115272 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:31:50.115887 (+ 615us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:31:50.118107 (+ 2220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:31:50.118113 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:31:50.118115 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:31:50.118709 (+ 594us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:31:50.119372 (+ 663us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:31:50.119377 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:31:50.119380 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:31:50.119554 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:31:50.120211 (+ 657us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:31:50.120217 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:31:50.120433 (+ 216us) client_negotiation.cc:769] Sending connection context
0812 02:31:50.120574 (+ 141us) client_negotiation.cc:240] Negotiation successful
0812 02:31:50.120771 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":616,"mutex_wait_us":39,"thread_start_us":166,"threads_started":1}
WARNING: no policy specified for kudu/127.14.103.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.103.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.103.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.2.keytab.
Entry for principal kudu/127.14.103.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.2.keytab.
I20260812 02:31:50.148139 14748 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbep4M4/build/debug/bin/kudu
/tmp/dist-test-taskbep4M4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501898713699-14748-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501898713699-14748-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501898713699-14748-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501898713699-14748-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.103.2:0
--local_ip_for_outbound_sockets=127.14.103.2
--webserver_interface=127.14.103.2
--webserver_port=0
--tserver_master_addrs=127.14.103.62:43757
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.2.keytab
--principal=kudu/127.14.103.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.103.20:37045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501898713699-14748-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501898713699-14748-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501898713699-14748-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:31:50.292804 16895 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:31:50.293172 16895 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:31:50.293228 16895 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:31:50.299058 16895 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:31:50.299167 16895 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:31:50.299218 16895 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260812 02:31:50.299233 16895 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 02:31:50.299269 16895 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.103.2
I20260812 02:31:50.306937 16895 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.103.20:37045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501898713699-14748-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501898713699-14748-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.2.keytab
--principal=kudu/127.14.103.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.103.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501898713699-14748-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.103.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.14.103.62:43757
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16895
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.103.2
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501898713699-14748-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:31:50.308347 16895 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:31:50.309607 16895 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:31:50.324795 16900 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:31:50.324795 16903 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:31:50.324865 16901 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:31:50.325989 16895 server_base.cc:1034] running on GCE node
I20260812 02:31:50.326599 16895 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:31:50.327512 16895 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:31:50.328748 16895 hybrid_clock.cc:648] HybridClock initialized: now 1786501910328703 us; error 66 us; skew 500 ppm
Aug 12 02:31:50 dist-test-slave-2w3w krb5kdc[16605](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501910, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:31:50.332497 16895 init.cc:377] Logged in from keytab as kudu/127.14.103.2@KRBTEST.COM (short username kudu)
I20260812 02:31:50.333618 16895 webserver.cc:460] Webserver started at http://127.14.103.2:33691/ using document root <none> and password file <none>
I20260812 02:31:50.334622 16895 fs_manager.cc:359] Metadata directory not provided
I20260812 02:31:50.334802 16895 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:31:50.335493 16895 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:31:50.338434 16895 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501898713699-14748-0/minicluster-data/ts-1/data/instance:
uuid: "f6f4b69dba6341ff926f2501f7d96928"
format_stamp: "Formatted at 2026-08-12 02:31:50 on dist-test-slave-2w3w"
server_key: "84bee916cfb2cd23e0a2b11a129c9e8c"
server_key_iv: "428ded626d6f9dfd2f5a643a15a167bc"
server_key_version: "encryptionkey@0"
I20260812 02:31:50.339223 16895 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501898713699-14748-0/minicluster-data/ts-1/wal/instance:
uuid: "f6f4b69dba6341ff926f2501f7d96928"
format_stamp: "Formatted at 2026-08-12 02:31:50 on dist-test-slave-2w3w"
server_key: "84bee916cfb2cd23e0a2b11a129c9e8c"
server_key_iv: "428ded626d6f9dfd2f5a643a15a167bc"
server_key_version: "encryptionkey@0"
I20260812 02:31:50.345465 16895 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260812 02:31:50.348744 16912 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:31:50.350586 16895 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260812 02:31:50.350766 16895 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501898713699-14748-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501898713699-14748-0/minicluster-data/ts-1/wal
uuid: "f6f4b69dba6341ff926f2501f7d96928"
format_stamp: "Formatted at 2026-08-12 02:31:50 on dist-test-slave-2w3w"
server_key: "84bee916cfb2cd23e0a2b11a129c9e8c"
server_key_iv: "428ded626d6f9dfd2f5a643a15a167bc"
server_key_version: "encryptionkey@0"
I20260812 02:31:50.350919 16895 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501898713699-14748-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501898713699-14748-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501898713699-14748-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:31:50.364782 16895 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:31:50.370201 16895 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:31:50.370437 16895 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:31:50.372325 16895 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:31:50.372416 16895 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:31:50.372495 16895 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:31:50.372530 16895 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 12 02:31:50 dist-test-slave-2w3w krb5kdc[16605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501910, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.2@KRBTEST.COM for kudu/127.14.103.62@KRBTEST.COM
I20260812 02:31:50.386473 16895 rpc_server.cc:307] RPC server started. Bound to: 127.14.103.2:46057
I20260812 02:31:50.386536 17037 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.103.2:46057 every 8 connection(s)
I20260812 02:31:50.387588 16895 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501898713699-14748-0/minicluster-data/ts-1/data/info.pb
I20260812 02:31:50.389492 14748 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbep4M4/build/debug/bin/kudu as pid 16895
I20260812 02:31:50.389626 14748 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501898713699-14748-0/minicluster-data/ts-1/wal/instance
I20260812 02:31:50.390019 14748 external_mini_cluster.cc:1428] Setting key ae94c33ce598e709ca889b3038b6b4a6
I20260812 02:31:50.391666 16718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:31:50.373362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.2:34629 (local address 127.14.103.62:43757)
0812 02:31:50.373543 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0812 02:31:50.373548 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:31:50.374590 (+ 1042us) server_negotiation.cc:407] Connection header received
0812 02:31:50.376848 (+ 2258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:31:50.376853 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:31:50.376936 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:31:50.377047 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:31:50.378833 (+ 1786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:31:50.380023 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:31:50.381631 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:31:50.381880 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:31:50.382024 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:31:50.386087 (+ 4063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:31:50.386092 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:31:50.386095 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:31:50.386200 (+ 105us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:31:50.388233 (+ 2033us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:31:50.389263 (+ 1030us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:31:50.389267 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:31:50.389269 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:31:50.389376 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:31:50.389950 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:31:50.389953 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:31:50.389954 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:31:50.390501 (+ 547us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:31:50.390659 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0812 02:31:50.391315 (+ 656us) server_negotiation.cc:299] Negotiation successful
0812 02:31:50.391496 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260812 02:31:50.392957 16922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:31:50.373851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:43757 (local address 127.14.103.2:34629)
0812 02:31:50.374368 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0812 02:31:50.374427 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0812 02:31:50.376506 (+ 2079us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:31:50.377371 (+ 865us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:31:50.377381 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:31:50.377979 (+ 598us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:31:50.378604 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:31:50.378621 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:31:50.380207 (+ 1586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:31:50.380212 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:31:50.381437 (+ 1225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:31:50.381451 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:31:50.382100 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:31:50.382104 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:31:50.382372 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:31:50.383843 (+ 1471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:31:50.383863 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:31:50.385833 (+ 1970us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:31:50.388438 (+ 2605us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:31:50.388445 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:31:50.388460 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:31:50.388802 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:31:50.389543 (+ 741us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:31:50.389550 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:31:50.389555 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:31:50.389810 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:31:50.390647 (+ 837us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:31:50.390654 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:31:50.391024 (+ 370us) client_negotiation.cc:769] Sending connection context
0812 02:31:50.391237 (+ 213us) client_negotiation.cc:240] Negotiation successful
0812 02:31:50.391685 (+ 448us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":138,"threads_started":1}
I20260812 02:31:50.395423 17038 heartbeater.cc:347] Connected to a master server at 127.14.103.62:43757
I20260812 02:31:50.395613 17038 heartbeater.cc:464] Registering TS with master...
I20260812 02:31:50.396363 17038 heartbeater.cc:511] Master 127.14.103.62:43757 requested a full tablet report, sending...
I20260812 02:31:50.397850 16664 ts_manager.cc:194] Registered new tserver with Master: f6f4b69dba6341ff926f2501f7d96928 (127.14.103.2:46057)
I20260812 02:31:50.399024 16664 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.103.2@KRBTEST.COM'} at 127.14.103.2:34629
WARNING: no policy specified for kudu/127.14.103.3@KRBTEST.COM; defaulting to no policy
I20260812 02:31:50.408125 16718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:31:50.398279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.2:54763 (local address 127.14.103.62:43757)
0812 02:31:50.398656 (+ 377us) server_negotiation.cc:206] Beginning negotiation
0812 02:31:50.398663 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 02:31:50.398757 (+ 94us) server_negotiation.cc:407] Connection header received
0812 02:31:50.398939 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:31:50.398944 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:31:50.399063 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:31:50.399219 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:31:50.400112 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:31:50.400948 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:31:50.402296 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:31:50.402570 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:31:50.402698 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:31:50.404164 (+ 1466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:31:50.404173 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:31:50.404177 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:31:50.404209 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:31:50.406003 (+ 1794us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:31:50.406518 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:31:50.406522 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:31:50.406524 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:31:50.406613 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:31:50.406998 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:31:50.407003 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:31:50.407006 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:31:50.407341 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:31:50.407461 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0812 02:31:50.407741 (+ 280us) server_negotiation.cc:299] Negotiation successful
0812 02:31:50.407955 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":244}
I20260812 02:31:50.408170 17045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:31:50.398213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:43757 (local address 127.14.103.2:54763)
0812 02:31:50.398658 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0812 02:31:50.398677 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 02:31:50.398803 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:31:50.399308 (+ 505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:31:50.399314 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:31:50.399660 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:31:50.399855 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:31:50.399868 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:31:50.401276 (+ 1408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:31:50.401283 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:31:50.402139 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:31:50.402151 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:31:50.402699 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:31:50.402703 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:31:50.402795 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:31:50.403512 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:31:50.403525 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:31:50.403886 (+ 361us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:31:50.406143 (+ 2257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:31:50.406147 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:31:50.406149 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:31:50.406406 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:31:50.406735 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:31:50.406738 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:31:50.406745 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:31:50.406831 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:31:50.407449 (+ 618us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:31:50.407452 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:31:50.407564 (+ 112us) client_negotiation.cc:769] Sending connection context
0812 02:31:50.407672 (+ 108us) client_negotiation.cc:240] Negotiation successful
0812 02:31:50.407869 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":134,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.103.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.103.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.3.keytab.
Entry for principal kudu/127.14.103.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.3.keytab.
I20260812 02:31:50.432219 14748 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbep4M4/build/debug/bin/kudu
/tmp/dist-test-taskbep4M4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501898713699-14748-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501898713699-14748-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501898713699-14748-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501898713699-14748-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.103.3:0
--local_ip_for_outbound_sockets=127.14.103.3
--webserver_interface=127.14.103.3
--webserver_port=0
--tserver_master_addrs=127.14.103.62:43757
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.3.keytab
--principal=kudu/127.14.103.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.103.20:37045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501898713699-14748-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501898713699-14748-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501898713699-14748-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:31:50.566332 17047 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:31:50.566716 17047 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:31:50.566771 17047 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:31:50.571512 17047 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:31:50.571596 17047 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:31:50.571669 17047 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260812 02:31:50.571698 17047 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 02:31:50.571761 17047 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.103.3
I20260812 02:31:50.578258 17047 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.103.20:37045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501898713699-14748-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501898713699-14748-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.3.keytab
--principal=kudu/127.14.103.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.103.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501898713699-14748-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.103.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.14.103.62:43757
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17047
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.103.3
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501898713699-14748-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:31:50.579631 17047 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:31:50.580809 17047 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:31:50.591967 17056 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:31:50.592000 17052 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:31:50.592149 17047 server_base.cc:1034] running on GCE node
W20260812 02:31:50.592497 17054 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:31:50.592950 17047 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:31:50.593742 17047 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:31:50.594954 17047 hybrid_clock.cc:648] HybridClock initialized: now 1786501910594893 us; error 77 us; skew 500 ppm
Aug 12 02:31:50 dist-test-slave-2w3w krb5kdc[16605](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501910, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:31:50.599853 17047 init.cc:377] Logged in from keytab as kudu/127.14.103.3@KRBTEST.COM (short username kudu)
I20260812 02:31:50.602028 17047 webserver.cc:460] Webserver started at http://127.14.103.3:43025/ using document root <none> and password file <none>
I20260812 02:31:50.602785 17047 fs_manager.cc:359] Metadata directory not provided
I20260812 02:31:50.602897 17047 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:31:50.603178 17047 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:31:50.605701 17047 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501898713699-14748-0/minicluster-data/ts-2/data/instance:
uuid: "063967dd04ee4fec97fccb5c6a5815f8"
format_stamp: "Formatted at 2026-08-12 02:31:50 on dist-test-slave-2w3w"
server_key: "330e80fa0812b41cdf4b787de9222a11"
server_key_iv: "6655fa81cee1709dafbad47b8fdcbdcf"
server_key_version: "encryptionkey@0"
I20260812 02:31:50.606532 17047 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501898713699-14748-0/minicluster-data/ts-2/wal/instance:
uuid: "063967dd04ee4fec97fccb5c6a5815f8"
format_stamp: "Formatted at 2026-08-12 02:31:50 on dist-test-slave-2w3w"
server_key: "330e80fa0812b41cdf4b787de9222a11"
server_key_iv: "6655fa81cee1709dafbad47b8fdcbdcf"
server_key_version: "encryptionkey@0"
I20260812 02:31:50.611359 17047 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.008s sys 0.000s
I20260812 02:31:50.614472 17064 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:31:50.616209 17047 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 02:31:50.616410 17047 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501898713699-14748-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501898713699-14748-0/minicluster-data/ts-2/wal
uuid: "063967dd04ee4fec97fccb5c6a5815f8"
format_stamp: "Formatted at 2026-08-12 02:31:50 on dist-test-slave-2w3w"
server_key: "330e80fa0812b41cdf4b787de9222a11"
server_key_iv: "6655fa81cee1709dafbad47b8fdcbdcf"
server_key_version: "encryptionkey@0"
I20260812 02:31:50.616585 17047 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501898713699-14748-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501898713699-14748-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501898713699-14748-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:31:50.657229 17047 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:31:50.663601 17047 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:31:50.663950 17047 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:31:50.665952 17047 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:31:50.666059 17047 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:31:50.666131 17047 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:31:50.666172 17047 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 12 02:31:50 dist-test-slave-2w3w krb5kdc[16605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501910, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.3@KRBTEST.COM for kudu/127.14.103.62@KRBTEST.COM
I20260812 02:31:50.678489 17047 rpc_server.cc:307] RPC server started. Bound to: 127.14.103.3:33411
I20260812 02:31:50.678613 17195 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.103.3:33411 every 8 connection(s)
I20260812 02:31:50.680182 17047 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501898713699-14748-0/minicluster-data/ts-2/data/info.pb
I20260812 02:31:50.683750 16718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:31:50.667342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.3:54413 (local address 127.14.103.62:43757)
0812 02:31:50.667511 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0812 02:31:50.667516 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:31:50.668783 (+ 1267us) server_negotiation.cc:407] Connection header received
0812 02:31:50.670561 (+ 1778us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:31:50.670567 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:31:50.670649 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:31:50.670752 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:31:50.672872 (+ 2120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:31:50.673632 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:31:50.675033 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:31:50.675290 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:31:50.675407 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:31:50.678401 (+ 2994us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:31:50.678404 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:31:50.678407 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:31:50.678465 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:31:50.680847 (+ 2382us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:31:50.681748 (+ 901us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:31:50.681752 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:31:50.681754 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:31:50.681894 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:31:50.682460 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:31:50.682463 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:31:50.682464 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:31:50.682687 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:31:50.682776 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0812 02:31:50.683474 (+ 698us) server_negotiation.cc:299] Negotiation successful
0812 02:31:50.683611 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260812 02:31:50.684624 17073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:31:50.668005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:43757 (local address 127.14.103.3:54413)
0812 02:31:50.668571 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0812 02:31:50.668642 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0812 02:31:50.670160 (+ 1518us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:31:50.671065 (+ 905us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:31:50.671080 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:31:50.671754 (+ 674us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:31:50.672639 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:31:50.672667 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:31:50.673793 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:31:50.673797 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:31:50.674839 (+ 1042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:31:50.674852 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:31:50.675474 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:31:50.675478 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:31:50.675684 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:31:50.676398 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:31:50.676416 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:31:50.678153 (+ 1737us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:31:50.681056 (+ 2903us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:31:50.681062 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:31:50.681079 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:31:50.681613 (+ 534us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:31:50.682050 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:31:50.682054 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:31:50.682056 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:31:50.682332 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:31:50.682835 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:31:50.682841 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:31:50.683196 (+ 355us) client_negotiation.cc:769] Sending connection context
0812 02:31:50.683374 (+ 178us) client_negotiation.cc:240] Negotiation successful
0812 02:31:50.683688 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":119,"threads_started":1}
I20260812 02:31:50.686723 17196 heartbeater.cc:347] Connected to a master server at 127.14.103.62:43757
I20260812 02:31:50.686952 17196 heartbeater.cc:464] Registering TS with master...
I20260812 02:31:50.687608 17196 heartbeater.cc:511] Master 127.14.103.62:43757 requested a full tablet report, sending...
I20260812 02:31:50.687829 14748 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbep4M4/build/debug/bin/kudu as pid 17047
I20260812 02:31:50.687960 14748 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501898713699-14748-0/minicluster-data/ts-2/wal/instance
I20260812 02:31:50.688453 14748 external_mini_cluster.cc:1428] Setting key 1924aad022389e36f5615257c308003b
I20260812 02:31:50.689316 16663 ts_manager.cc:194] Registered new tserver with Master: 063967dd04ee4fec97fccb5c6a5815f8 (127.14.103.3:33411)
I20260812 02:31:50.691069 16663 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.103.3@KRBTEST.COM'} at 127.14.103.3:54413
I20260812 02:31:50.691299 14748 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 02:31:50.699787 16718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:31:50.689278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.3:34641 (local address 127.14.103.62:43757)
0812 02:31:50.689432 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0812 02:31:50.689437 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:31:50.690577 (+ 1140us) server_negotiation.cc:407] Connection header received
0812 02:31:50.690637 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:31:50.690641 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:31:50.690691 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:31:50.690797 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:31:50.691505 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:31:50.692151 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:31:50.693010 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:31:50.693252 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:31:50.693375 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:31:50.694921 (+ 1546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:31:50.694928 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:31:50.694932 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:31:50.695002 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:31:50.697016 (+ 2014us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:31:50.697769 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:31:50.697772 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:31:50.697773 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:31:50.697876 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:31:50.698464 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:31:50.698467 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:31:50.698468 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:31:50.698708 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:31:50.698805 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0812 02:31:50.699509 (+ 704us) server_negotiation.cc:299] Negotiation successful
0812 02:31:50.699648 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260812 02:31:50.700333 17204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:31:50.689487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:43757 (local address 127.14.103.3:34641)
0812 02:31:50.690106 (+ 619us) negotiation.cc:107] Waiting for socket to connect
0812 02:31:50.690124 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 02:31:50.690287 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:31:50.690854 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:31:50.690857 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:31:50.691180 (+ 323us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:31:50.691331 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:31:50.691341 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:31:50.692286 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:31:50.692290 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:31:50.692884 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:31:50.692892 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:31:50.693411 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:31:50.693415 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:31:50.693531 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:31:50.694250 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:31:50.694270 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:31:50.694716 (+ 446us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:31:50.697263 (+ 2547us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:31:50.697270 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:31:50.697276 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:31:50.697623 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:31:50.697991 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:31:50.697997 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:31:50.698002 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:31:50.698212 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:31:50.698846 (+ 634us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:31:50.698850 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:31:50.699127 (+ 277us) client_negotiation.cc:769] Sending connection context
0812 02:31:50.699455 (+ 328us) client_negotiation.cc:240] Negotiation successful
0812 02:31:50.699733 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":368,"mutex_wait_us":20,"thread_start_us":116,"threads_started":1}
Aug 12 02:31:50 dist-test-slave-2w3w krb5kdc[16605](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501910, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 12 02:31:50 dist-test-slave-2w3w krb5kdc[16605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501910, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.103.62@KRBTEST.COM
I20260812 02:31:50.734460 16718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:31:50.720366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51182 (local address 127.14.103.62:43757)
0812 02:31:50.720518 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0812 02:31:50.720522 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:31:50.720557 (+ 35us) server_negotiation.cc:407] Connection header received
0812 02:31:50.720618 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:31:50.720624 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:31:50.720683 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:31:50.720789 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:31:50.721706 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:31:50.722363 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:31:50.723786 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:31:50.724058 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:31:50.724151 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:31:50.728410 (+ 4259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:31:50.728414 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:31:50.728416 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:31:50.728458 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:31:50.730887 (+ 2429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:31:50.732372 (+ 1485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:31:50.732382 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:31:50.732383 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:31:50.732523 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:31:50.733218 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:31:50.733223 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:31:50.733226 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:31:50.733550 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:31:50.733692 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0812 02:31:50.734093 (+ 401us) server_negotiation.cc:299] Negotiation successful
0812 02:31:50.734305 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260812 02:31:50.741370 16663 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51182:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 02:31:50.750223 16663 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 02:31:50.771986 17221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:31:50.764944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.1:38185 (local address 127.0.0.1:38290)
0812 02:31:50.765628 (+ 684us) negotiation.cc:107] Waiting for socket to connect
0812 02:31:50.765642 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 02:31:50.765736 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:31:50.766687 (+ 951us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:31:50.766691 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:31:50.766713 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:31:50.766815 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:31:50.766821 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:31:50.768190 (+ 1369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:31:50.768195 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:31:50.770184 (+ 1989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:31:50.770197 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:31:50.771632 (+ 1435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:31:50.771635 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:31:50.771720 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:31:50.771739 (+ 19us) client_negotiation.cc:769] Sending connection context
0812 02:31:50.771803 (+ 64us) client_negotiation.cc:240] Negotiation successful
0812 02:31:50.771859 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":611,"mutex_wait_us":71,"thread_start_us":112,"threads_started":1}
I20260812 02:31:50.771984 17219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:31:50.764688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.3:33411 (local address 127.0.0.1:57520)
0812 02:31:50.765225 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0812 02:31:50.765290 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0812 02:31:50.765443 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:31:50.766293 (+ 850us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:31:50.766300 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:31:50.766324 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:31:50.766512 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:31:50.766521 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:31:50.768066 (+ 1545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:31:50.768069 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:31:50.770064 (+ 1995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:31:50.770075 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:31:50.771309 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:31:50.771313 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:31:50.771448 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:31:50.771505 (+ 57us) client_negotiation.cc:769] Sending connection context
0812 02:31:50.771636 (+ 131us) client_negotiation.cc:240] Negotiation successful
0812 02:31:50.771787 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":334,"mutex_wait_us":1,"thread_start_us":166,"threads_started":1}
I20260812 02:31:50.772277 17220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:31:50.764834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57520 (local address 127.14.103.3:33411)
0812 02:31:50.765334 (+ 500us) server_negotiation.cc:206] Beginning negotiation
0812 02:31:50.765350 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0812 02:31:50.765379 (+ 29us) server_negotiation.cc:407] Connection header received
0812 02:31:50.765687 (+ 308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:31:50.765694 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:31:50.765927 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:31:50.766110 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:31:50.766653 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:31:50.767938 (+ 1285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:31:50.770224 (+ 2286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:31:50.771192 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:31:50.771318 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:31:50.771524 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0812 02:31:50.771683 (+ 159us) server_negotiation.cc:299] Negotiation successful
0812 02:31:50.771811 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":398,"thread_start_us":164,"threads_started":1}
I20260812 02:31:50.773108 17222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:31:50.765669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.2:46057 (local address 127.0.0.1:55176)
0812 02:31:50.766549 (+ 880us) negotiation.cc:107] Waiting for socket to connect
0812 02:31:50.766565 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 02:31:50.766657 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:31:50.767653 (+ 996us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:31:50.767656 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:31:50.767678 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:31:50.767824 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:31:50.767831 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:31:50.769767 (+ 1936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:31:50.769770 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:31:50.771130 (+ 1360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:31:50.771143 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:31:50.772665 (+ 1522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:31:50.772669 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:31:50.772774 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:31:50.772796 (+ 22us) client_negotiation.cc:769] Sending connection context
0812 02:31:50.772880 (+ 84us) client_negotiation.cc:240] Negotiation successful
0812 02:31:50.772963 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":818,"mutex_wait_us":7,"thread_start_us":112,"threads_started":1}
I20260812 02:31:50.773131 17224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:31:50.766152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55176 (local address 127.14.103.2:46057)
0812 02:31:50.767051 (+ 899us) server_negotiation.cc:206] Beginning negotiation
0812 02:31:50.767075 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0812 02:31:50.767107 (+ 32us) server_negotiation.cc:407] Connection header received
0812 02:31:50.767235 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:31:50.767241 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:31:50.767402 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:31:50.767661 (+ 259us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:31:50.767972 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:31:50.769441 (+ 1469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:31:50.771315 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:31:50.772522 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:31:50.772665 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:31:50.772848 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0812 02:31:50.772906 (+ 58us) server_negotiation.cc:299] Negotiation successful
0812 02:31:50.772997 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":669,"thread_start_us":123,"threads_started":1}
I20260812 02:31:50.773553 17223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:31:50.765697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38290 (local address 127.14.103.1:38185)
0812 02:31:50.766087 (+ 390us) server_negotiation.cc:206] Beginning negotiation
0812 02:31:50.766094 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 02:31:50.766115 (+ 21us) server_negotiation.cc:407] Connection header received
0812 02:31:50.766190 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:31:50.766195 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:31:50.766371 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:31:50.766536 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:31:50.766950 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:31:50.768066 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:31:50.770349 (+ 2283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:31:50.771308 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:31:50.771413 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:31:50.771565 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0812 02:31:50.773292 (+ 1727us) server_negotiation.cc:299] Negotiation successful
0812 02:31:50.773411 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":99,"threads_started":1}
I20260812 02:31:50.775532 17118 tablet_service.cc:1467] Processing CreateTablet for tablet 4440a6466c6643d0bfffee353db96a25 (DEFAULT_TABLE table=test-table [id=1fb79426247e406eb08e908e72658b2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:31:50.776161 16814 tablet_service.cc:1467] Processing CreateTablet for tablet 4440a6466c6643d0bfffee353db96a25 (DEFAULT_TABLE table=test-table [id=1fb79426247e406eb08e908e72658b2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:31:50.777037 17118 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4440a6466c6643d0bfffee353db96a25. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:31:50.777206 16814 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4440a6466c6643d0bfffee353db96a25. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:31:50.777890 16966 tablet_service.cc:1467] Processing CreateTablet for tablet 4440a6466c6643d0bfffee353db96a25 (DEFAULT_TABLE table=test-table [id=1fb79426247e406eb08e908e72658b2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:31:50.779088 16966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4440a6466c6643d0bfffee353db96a25. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:31:50.784106 17225 tablet_bootstrap.cc:492] T 4440a6466c6643d0bfffee353db96a25 P ac78ac69480a4042a6278f187e3b4d61: Bootstrap starting.
I20260812 02:31:50.786746 17225 tablet_bootstrap.cc:654] T 4440a6466c6643d0bfffee353db96a25 P ac78ac69480a4042a6278f187e3b4d61: Neither blocks nor log segments found. Creating new log.
I20260812 02:31:50.786849 17226 tablet_bootstrap.cc:492] T 4440a6466c6643d0bfffee353db96a25 P 063967dd04ee4fec97fccb5c6a5815f8: Bootstrap starting.
I20260812 02:31:50.787874 17225 log.cc:824] T 4440a6466c6643d0bfffee353db96a25 P ac78ac69480a4042a6278f187e3b4d61: Log is configured to *not* fsync() on all Append() calls
I20260812 02:31:50.788893 17227 tablet_bootstrap.cc:492] T 4440a6466c6643d0bfffee353db96a25 P f6f4b69dba6341ff926f2501f7d96928: Bootstrap starting.
I20260812 02:31:50.789259 17226 tablet_bootstrap.cc:654] T 4440a6466c6643d0bfffee353db96a25 P 063967dd04ee4fec97fccb5c6a5815f8: Neither blocks nor log segments found. Creating new log.
I20260812 02:31:50.790196 17225 tablet_bootstrap.cc:492] T 4440a6466c6643d0bfffee353db96a25 P ac78ac69480a4042a6278f187e3b4d61: No bootstrap required, opened a new log
I20260812 02:31:50.790144 17226 log.cc:824] T 4440a6466c6643d0bfffee353db96a25 P 063967dd04ee4fec97fccb5c6a5815f8: Log is configured to *not* fsync() on all Append() calls
I20260812 02:31:50.790421 17225 ts_tablet_manager.cc:1397] T 4440a6466c6643d0bfffee353db96a25 P ac78ac69480a4042a6278f187e3b4d61: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260812 02:31:50.791750 17227 tablet_bootstrap.cc:654] T 4440a6466c6643d0bfffee353db96a25 P f6f4b69dba6341ff926f2501f7d96928: Neither blocks nor log segments found. Creating new log.
I20260812 02:31:50.792873 17227 log.cc:824] T 4440a6466c6643d0bfffee353db96a25 P f6f4b69dba6341ff926f2501f7d96928: Log is configured to *not* fsync() on all Append() calls
I20260812 02:31:50.792976 17226 tablet_bootstrap.cc:492] T 4440a6466c6643d0bfffee353db96a25 P 063967dd04ee4fec97fccb5c6a5815f8: No bootstrap required, opened a new log
I20260812 02:31:50.793205 17226 ts_tablet_manager.cc:1397] T 4440a6466c6643d0bfffee353db96a25 P 063967dd04ee4fec97fccb5c6a5815f8: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260812 02:31:50.795156 17225 raft_consensus.cc:348] T 4440a6466c6643d0bfffee353db96a25 P ac78ac69480a4042a6278f187e3b4d61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "063967dd04ee4fec97fccb5c6a5815f8" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 33411 } } peers { permanent_uuid: "ac78ac69480a4042a6278f187e3b4d61" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 38185 } } peers { permanent_uuid: "f6f4b69dba6341ff926f2501f7d96928" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 46057 } }
I20260812 02:31:50.795456 17225 raft_consensus.cc:374] T 4440a6466c6643d0bfffee353db96a25 P ac78ac69480a4042a6278f187e3b4d61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:31:50.795519 17225 raft_consensus.cc:729] T 4440a6466c6643d0bfffee353db96a25 P ac78ac69480a4042a6278f187e3b4d61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac78ac69480a4042a6278f187e3b4d61, State: Initialized, Role: FOLLOWER
I20260812 02:31:50.795967 17227 tablet_bootstrap.cc:492] T 4440a6466c6643d0bfffee353db96a25 P f6f4b69dba6341ff926f2501f7d96928: No bootstrap required, opened a new log
I20260812 02:31:50.795982 17225 consensus_queue.cc:260] T 4440a6466c6643d0bfffee353db96a25 P ac78ac69480a4042a6278f187e3b4d61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "063967dd04ee4fec97fccb5c6a5815f8" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 33411 } } peers { permanent_uuid: "ac78ac69480a4042a6278f187e3b4d61" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 38185 } } peers { permanent_uuid: "f6f4b69dba6341ff926f2501f7d96928" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 46057 } }
I20260812 02:31:50.796221 17227 ts_tablet_manager.cc:1397] T 4440a6466c6643d0bfffee353db96a25 P f6f4b69dba6341ff926f2501f7d96928: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260812 02:31:50.796789 16885 heartbeater.cc:503] Master 127.14.103.62:43757 was elected leader, sending a full tablet report...
I20260812 02:31:50.797430 17225 ts_tablet_manager.cc:1428] T 4440a6466c6643d0bfffee353db96a25 P ac78ac69480a4042a6278f187e3b4d61: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260812 02:31:50.797940 17226 raft_consensus.cc:348] T 4440a6466c6643d0bfffee353db96a25 P 063967dd04ee4fec97fccb5c6a5815f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "063967dd04ee4fec97fccb5c6a5815f8" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 33411 } } peers { permanent_uuid: "ac78ac69480a4042a6278f187e3b4d61" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 38185 } } peers { permanent_uuid: "f6f4b69dba6341ff926f2501f7d96928" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 46057 } }
I20260812 02:31:50.798384 17226 raft_consensus.cc:374] T 4440a6466c6643d0bfffee353db96a25 P 063967dd04ee4fec97fccb5c6a5815f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:31:50.798470 17226 raft_consensus.cc:729] T 4440a6466c6643d0bfffee353db96a25 P 063967dd04ee4fec97fccb5c6a5815f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 063967dd04ee4fec97fccb5c6a5815f8, State: Initialized, Role: FOLLOWER
I20260812 02:31:50.799039 17226 consensus_queue.cc:260] T 4440a6466c6643d0bfffee353db96a25 P 063967dd04ee4fec97fccb5c6a5815f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "063967dd04ee4fec97fccb5c6a5815f8" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 33411 } } peers { permanent_uuid: "ac78ac69480a4042a6278f187e3b4d61" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 38185 } } peers { permanent_uuid: "f6f4b69dba6341ff926f2501f7d96928" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 46057 } }
I20260812 02:31:50.799880 17196 heartbeater.cc:503] Master 127.14.103.62:43757 was elected leader, sending a full tablet report...
I20260812 02:31:50.800251 17226 ts_tablet_manager.cc:1428] T 4440a6466c6643d0bfffee353db96a25 P 063967dd04ee4fec97fccb5c6a5815f8: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260812 02:31:50.801115 17227 raft_consensus.cc:348] T 4440a6466c6643d0bfffee353db96a25 P f6f4b69dba6341ff926f2501f7d96928 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "063967dd04ee4fec97fccb5c6a5815f8" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 33411 } } peers { permanent_uuid: "ac78ac69480a4042a6278f187e3b4d61" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 38185 } } peers { permanent_uuid: "f6f4b69dba6341ff926f2501f7d96928" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 46057 } }
I20260812 02:31:50.801448 17227 raft_consensus.cc:374] T 4440a6466c6643d0bfffee353db96a25 P f6f4b69dba6341ff926f2501f7d96928 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:31:50.801508 17227 raft_consensus.cc:729] T 4440a6466c6643d0bfffee353db96a25 P f6f4b69dba6341ff926f2501f7d96928 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6f4b69dba6341ff926f2501f7d96928, State: Initialized, Role: FOLLOWER
I20260812 02:31:50.801963 17227 consensus_queue.cc:260] T 4440a6466c6643d0bfffee353db96a25 P f6f4b69dba6341ff926f2501f7d96928 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "063967dd04ee4fec97fccb5c6a5815f8" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 33411 } } peers { permanent_uuid: "ac78ac69480a4042a6278f187e3b4d61" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 38185 } } peers { permanent_uuid: "f6f4b69dba6341ff926f2501f7d96928" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 46057 } }
I20260812 02:31:50.803485 17227 ts_tablet_manager.cc:1428] T 4440a6466c6643d0bfffee353db96a25 P f6f4b69dba6341ff926f2501f7d96928: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260812 02:31:50.803900 17038 heartbeater.cc:503] Master 127.14.103.62:43757 was elected leader, sending a full tablet report...
W20260812 02:31:50.850646 16886 tablet.cc:2369] T 4440a6466c6643d0bfffee353db96a25 P ac78ac69480a4042a6278f187e3b4d61: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 02:31:50.889309 17039 tablet.cc:2369] T 4440a6466c6643d0bfffee353db96a25 P f6f4b69dba6341ff926f2501f7d96928: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 02:31:50.931911 17197 tablet.cc:2369] T 4440a6466c6643d0bfffee353db96a25 P 063967dd04ee4fec97fccb5c6a5815f8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 02:31:51.118554 17233 raft_consensus.cc:482] T 4440a6466c6643d0bfffee353db96a25 P f6f4b69dba6341ff926f2501f7d96928 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 02:31:51.118788 17233 raft_consensus.cc:504] T 4440a6466c6643d0bfffee353db96a25 P f6f4b69dba6341ff926f2501f7d96928 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "063967dd04ee4fec97fccb5c6a5815f8" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 33411 } } peers { permanent_uuid: "ac78ac69480a4042a6278f187e3b4d61" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 38185 } } peers { permanent_uuid: "f6f4b69dba6341ff926f2501f7d96928" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 46057 } }
I20260812 02:31:51.120185 17233 leader_election.cc:288] T 4440a6466c6643d0bfffee353db96a25 P f6f4b69dba6341ff926f2501f7d96928 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 063967dd04ee4fec97fccb5c6a5815f8 (127.14.103.3:33411), ac78ac69480a4042a6278f187e3b4d61 (127.14.103.1:38185)
I20260812 02:31:51.125126 17220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:31:51.120733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.2:49733 (local address 127.14.103.3:33411)
0812 02:31:51.120895 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0812 02:31:51.120902 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 02:31:51.120975 (+ 73us) server_negotiation.cc:407] Connection header received
0812 02:31:51.121365 (+ 390us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:31:51.121368 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:31:51.121410 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:31:51.121523 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:31:51.121850 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:31:51.122486 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:31:51.123955 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:31:51.124415 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:31:51.124522 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:31:51.124564 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0812 02:31:51.124841 (+ 277us) server_negotiation.cc:299] Negotiation successful
0812 02:31:51.124967 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260812 02:31:51.125101 17236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:31:51.120533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.3:33411 (local address 127.14.103.2:49733)
0812 02:31:51.120865 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0812 02:31:51.120877 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 02:31:51.121051 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:31:51.121549 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:31:51.121552 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:31:51.121570 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:31:51.121700 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:31:51.121705 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:31:51.122609 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:31:51.122612 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:31:51.123803 (+ 1191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:31:51.123814 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:31:51.124552 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:31:51.124556 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:31:51.124667 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:31:51.124697 (+ 30us) client_negotiation.cc:769] Sending connection context
0812 02:31:51.124787 (+ 90us) client_negotiation.cc:240] Negotiation successful
0812 02:31:51.124880 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"spinlock_wait_cycles":3328,"thread_start_us":87,"threads_started":1}
I20260812 02:31:51.125478 17234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:31:51.120477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.1:38185 (local address 127.14.103.2:49041)
0812 02:31:51.120813 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0812 02:31:51.120829 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 02:31:51.121053 (+ 224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:31:51.121742 (+ 689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:31:51.121743 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:31:51.121819 (+ 76us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:31:51.121949 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:31:51.121956 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:31:51.123166 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:31:51.123175 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:31:51.123973 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:31:51.123984 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:31:51.124855 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:31:51.124859 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:31:51.124964 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:31:51.124988 (+ 24us) client_negotiation.cc:769] Sending connection context
0812 02:31:51.125069 (+ 81us) client_negotiation.cc:240] Negotiation successful
0812 02:31:51.125142 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"spinlock_wait_cycles":36992,"thread_start_us":78,"threads_started":1}
I20260812 02:31:51.126041 17223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:31:51.120586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.2:49041 (local address 127.14.103.1:38185)
0812 02:31:51.120727 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0812 02:31:51.120732 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:31:51.120959 (+ 227us) server_negotiation.cc:407] Connection header received
0812 02:31:51.121362 (+ 403us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:31:51.121365 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:31:51.121411 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:31:51.121476 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:31:51.122200 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:31:51.122931 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:31:51.124192 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:31:51.124723 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:31:51.124849 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:31:51.124941 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0812 02:31:51.125196 (+ 255us) server_negotiation.cc:299] Negotiation successful
0812 02:31:51.125844 (+ 648us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260812 02:31:51.126037 17142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4440a6466c6643d0bfffee353db96a25" candidate_uuid: "f6f4b69dba6341ff926f2501f7d96928" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "063967dd04ee4fec97fccb5c6a5815f8" is_pre_election: true
I20260812 02:31:51.126513 17142 raft_consensus.cc:2454] T 4440a6466c6643d0bfffee353db96a25 P 063967dd04ee4fec97fccb5c6a5815f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6f4b69dba6341ff926f2501f7d96928 in term 0.
I20260812 02:31:51.126942 16836 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4440a6466c6643d0bfffee353db96a25" candidate_uuid: "f6f4b69dba6341ff926f2501f7d96928" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac78ac69480a4042a6278f187e3b4d61" is_pre_election: true
I20260812 02:31:51.127256 16914 leader_election.cc:302] T 4440a6466c6643d0bfffee353db96a25 P f6f4b69dba6341ff926f2501f7d96928 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 063967dd04ee4fec97fccb5c6a5815f8, f6f4b69dba6341ff926f2501f7d96928; no voters:
I20260812 02:31:51.127359 16836 raft_consensus.cc:2454] T 4440a6466c6643d0bfffee353db96a25 P ac78ac69480a4042a6278f187e3b4d61 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6f4b69dba6341ff926f2501f7d96928 in term 0.
I20260812 02:31:51.127622 17233 raft_consensus.cc:2781] T 4440a6466c6643d0bfffee353db96a25 P f6f4b69dba6341ff926f2501f7d96928 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 02:31:51.127737 17233 raft_consensus.cc:482] T 4440a6466c6643d0bfffee353db96a25 P f6f4b69dba6341ff926f2501f7d96928 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 02:31:51.127815 17233 raft_consensus.cc:3037] T 4440a6466c6643d0bfffee353db96a25 P f6f4b69dba6341ff926f2501f7d96928 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:31:51.129078 17233 raft_consensus.cc:504] T 4440a6466c6643d0bfffee353db96a25 P f6f4b69dba6341ff926f2501f7d96928 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "063967dd04ee4fec97fccb5c6a5815f8" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 33411 } } peers { permanent_uuid: "ac78ac69480a4042a6278f187e3b4d61" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 38185 } } peers { permanent_uuid: "f6f4b69dba6341ff926f2501f7d96928" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 46057 } }
I20260812 02:31:51.129587 17233 leader_election.cc:288] T 4440a6466c6643d0bfffee353db96a25 P f6f4b69dba6341ff926f2501f7d96928 [CANDIDATE]: Term 1 election: Requested vote from peers 063967dd04ee4fec97fccb5c6a5815f8 (127.14.103.3:33411), ac78ac69480a4042a6278f187e3b4d61 (127.14.103.1:38185)
I20260812 02:31:51.130084 16836 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4440a6466c6643d0bfffee353db96a25" candidate_uuid: "f6f4b69dba6341ff926f2501f7d96928" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac78ac69480a4042a6278f187e3b4d61"
I20260812 02:31:51.130084 17142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4440a6466c6643d0bfffee353db96a25" candidate_uuid: "f6f4b69dba6341ff926f2501f7d96928" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "063967dd04ee4fec97fccb5c6a5815f8"
I20260812 02:31:51.130262 16836 raft_consensus.cc:3037] T 4440a6466c6643d0bfffee353db96a25 P ac78ac69480a4042a6278f187e3b4d61 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:31:51.130252 17142 raft_consensus.cc:3037] T 4440a6466c6643d0bfffee353db96a25 P 063967dd04ee4fec97fccb5c6a5815f8 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:31:51.131742 16836 raft_consensus.cc:2454] T 4440a6466c6643d0bfffee353db96a25 P ac78ac69480a4042a6278f187e3b4d61 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6f4b69dba6341ff926f2501f7d96928 in term 1.
I20260812 02:31:51.131752 17142 raft_consensus.cc:2454] T 4440a6466c6643d0bfffee353db96a25 P 063967dd04ee4fec97fccb5c6a5815f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6f4b69dba6341ff926f2501f7d96928 in term 1.
I20260812 02:31:51.132122 16916 leader_election.cc:302] T 4440a6466c6643d0bfffee353db96a25 P f6f4b69dba6341ff926f2501f7d96928 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac78ac69480a4042a6278f187e3b4d61, f6f4b69dba6341ff926f2501f7d96928; no voters:
I20260812 02:31:51.132350 17233 raft_consensus.cc:2781] T 4440a6466c6643d0bfffee353db96a25 P f6f4b69dba6341ff926f2501f7d96928 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:31:51.132543 17233 raft_consensus.cc:686] T 4440a6466c6643d0bfffee353db96a25 P f6f4b69dba6341ff926f2501f7d96928 [term 1 LEADER]: Becoming Leader. State: Replica: f6f4b69dba6341ff926f2501f7d96928, State: Running, Role: LEADER
I20260812 02:31:51.132956 17233 consensus_queue.cc:237] T 4440a6466c6643d0bfffee353db96a25 P f6f4b69dba6341ff926f2501f7d96928 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "063967dd04ee4fec97fccb5c6a5815f8" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 33411 } } peers { permanent_uuid: "ac78ac69480a4042a6278f187e3b4d61" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 38185 } } peers { permanent_uuid: "f6f4b69dba6341ff926f2501f7d96928" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 46057 } }
I20260812 02:31:51.136549 16663 catalog_manager.cc:5495] T 4440a6466c6643d0bfffee353db96a25 P f6f4b69dba6341ff926f2501f7d96928 reported cstate change: term changed from 0 to 1, leader changed from <none> to f6f4b69dba6341ff926f2501f7d96928 (127.14.103.2). New cstate: current_term: 1 leader_uuid: "f6f4b69dba6341ff926f2501f7d96928" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "063967dd04ee4fec97fccb5c6a5815f8" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 33411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ac78ac69480a4042a6278f187e3b4d61" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 38185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f6f4b69dba6341ff926f2501f7d96928" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 46057 } health_report { overall_health: HEALTHY } } }
Aug 12 02:31:51 dist-test-slave-2w3w krb5kdc[16605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501909, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.62@KRBTEST.COM for kudu/127.14.103.62@KRBTEST.COM
I20260812 02:31:51.269590 17244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:31:51.257979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51184 (local address 127.14.103.62:43757)
0812 02:31:51.258529 (+ 550us) server_negotiation.cc:206] Beginning negotiation
0812 02:31:51.258541 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 02:31:51.258669 (+ 128us) server_negotiation.cc:407] Connection header received
0812 02:31:51.258906 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:31:51.258909 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:31:51.258981 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:31:51.259071 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:31:51.260099 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:31:51.261000 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:31:51.262109 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:31:51.262537 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:31:51.262645 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:31:51.265626 (+ 2981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:31:51.265630 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:31:51.265632 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:31:51.265655 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:31:51.267362 (+ 1707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:31:51.267929 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:31:51.267933 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:31:51.267935 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:31:51.268036 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:31:51.268436 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:31:51.268439 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:31:51.268440 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:31:51.268667 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:31:51.268851 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0812 02:31:51.269196 (+ 345us) server_negotiation.cc:299] Negotiation successful
0812 02:31:51.269380 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":429,"spinlock_wait_cycles":12672,"thread_start_us":201,"threads_started":1}
I20260812 02:31:51.269590 17245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:31:51.257971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:43757 (local address 127.0.0.1:51184)
0812 02:31:51.258527 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0812 02:31:51.258544 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 02:31:51.258677 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:31:51.259155 (+ 478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:31:51.259157 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:31:51.259689 (+ 532us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:31:51.259798 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:31:51.259805 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:31:51.261209 (+ 1404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:31:51.261212 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:31:51.261837 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:31:51.261847 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:31:51.262698 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:31:51.262701 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:31:51.262776 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:31:51.263757 (+ 981us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:31:51.263771 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:31:51.265450 (+ 1679us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:31:51.267522 (+ 2072us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:31:51.267534 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:31:51.267540 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:31:51.267810 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:31:51.268155 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:31:51.268159 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:31:51.268160 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:31:51.268255 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:31:51.268851 (+ 596us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:31:51.268909 (+ 58us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:31:51.269019 (+ 110us) client_negotiation.cc:769] Sending connection context
0812 02:31:51.269149 (+ 130us) client_negotiation.cc:240] Negotiation successful
0812 02:31:51.269447 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":440,"mutex_wait_us":127,"thread_start_us":315,"threads_started":1}
I20260812 02:31:51.275154 16663 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.14.103.62@KRBTEST.COM'} at 127.0.0.1:51184:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260812 02:31:51.275918 16663 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 02:31:51.286239 16814 tablet_service.cc:1467] Processing CreateTablet for tablet 16010c528ee0420b82f75881295ff1ef (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=16c80ed23c6b4350b5d83cd2839698c5]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260812 02:31:51.286176 16966 tablet_service.cc:1467] Processing CreateTablet for tablet 16010c528ee0420b82f75881295ff1ef (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=16c80ed23c6b4350b5d83cd2839698c5]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260812 02:31:51.286177 17118 tablet_service.cc:1467] Processing CreateTablet for tablet 16010c528ee0420b82f75881295ff1ef (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=16c80ed23c6b4350b5d83cd2839698c5]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260812 02:31:51.287072 16814 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16010c528ee0420b82f75881295ff1ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:31:51.287066 17118 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16010c528ee0420b82f75881295ff1ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:31:51.287066 16966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16010c528ee0420b82f75881295ff1ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:31:51.290542 17225 tablet_bootstrap.cc:492] T 16010c528ee0420b82f75881295ff1ef P ac78ac69480a4042a6278f187e3b4d61: Bootstrap starting.
I20260812 02:31:51.290553 17226 tablet_bootstrap.cc:492] T 16010c528ee0420b82f75881295ff1ef P 063967dd04ee4fec97fccb5c6a5815f8: Bootstrap starting.
I20260812 02:31:51.291747 17226 tablet_bootstrap.cc:654] T 16010c528ee0420b82f75881295ff1ef P 063967dd04ee4fec97fccb5c6a5815f8: Neither blocks nor log segments found. Creating new log.
I20260812 02:31:51.291772 17227 tablet_bootstrap.cc:492] T 16010c528ee0420b82f75881295ff1ef P f6f4b69dba6341ff926f2501f7d96928: Bootstrap starting.
I20260812 02:31:51.291877 17225 tablet_bootstrap.cc:654] T 16010c528ee0420b82f75881295ff1ef P ac78ac69480a4042a6278f187e3b4d61: Neither blocks nor log segments found. Creating new log.
I20260812 02:31:51.292980 17227 tablet_bootstrap.cc:654] T 16010c528ee0420b82f75881295ff1ef P f6f4b69dba6341ff926f2501f7d96928: Neither blocks nor log segments found. Creating new log.
I20260812 02:31:51.293395 17226 tablet_bootstrap.cc:492] T 16010c528ee0420b82f75881295ff1ef P 063967dd04ee4fec97fccb5c6a5815f8: No bootstrap required, opened a new log
I20260812 02:31:51.293534 17226 ts_tablet_manager.cc:1397] T 16010c528ee0420b82f75881295ff1ef P 063967dd04ee4fec97fccb5c6a5815f8: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260812 02:31:51.293630 17225 tablet_bootstrap.cc:492] T 16010c528ee0420b82f75881295ff1ef P ac78ac69480a4042a6278f187e3b4d61: No bootstrap required, opened a new log
I20260812 02:31:51.293732 17225 ts_tablet_manager.cc:1397] T 16010c528ee0420b82f75881295ff1ef P ac78ac69480a4042a6278f187e3b4d61: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260812 02:31:51.294305 17225 raft_consensus.cc:348] T 16010c528ee0420b82f75881295ff1ef P ac78ac69480a4042a6278f187e3b4d61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac78ac69480a4042a6278f187e3b4d61" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 38185 } } peers { permanent_uuid: "f6f4b69dba6341ff926f2501f7d96928" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 46057 } } peers { permanent_uuid: "063967dd04ee4fec97fccb5c6a5815f8" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 33411 } }
I20260812 02:31:51.294453 17225 raft_consensus.cc:374] T 16010c528ee0420b82f75881295ff1ef P ac78ac69480a4042a6278f187e3b4d61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:31:51.294497 17225 raft_consensus.cc:729] T 16010c528ee0420b82f75881295ff1ef P ac78ac69480a4042a6278f187e3b4d61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac78ac69480a4042a6278f187e3b4d61, State: Initialized, Role: FOLLOWER
I20260812 02:31:51.294461 17226 raft_consensus.cc:348] T 16010c528ee0420b82f75881295ff1ef P 063967dd04ee4fec97fccb5c6a5815f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac78ac69480a4042a6278f187e3b4d61" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 38185 } } peers { permanent_uuid: "f6f4b69dba6341ff926f2501f7d96928" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 46057 } } peers { permanent_uuid: "063967dd04ee4fec97fccb5c6a5815f8" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 33411 } }
I20260812 02:31:51.294605 17226 raft_consensus.cc:374] T 16010c528ee0420b82f75881295ff1ef P 063967dd04ee4fec97fccb5c6a5815f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:31:51.294644 17226 raft_consensus.cc:729] T 16010c528ee0420b82f75881295ff1ef P 063967dd04ee4fec97fccb5c6a5815f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 063967dd04ee4fec97fccb5c6a5815f8, State: Initialized, Role: FOLLOWER
I20260812 02:31:51.294639 17225 consensus_queue.cc:260] T 16010c528ee0420b82f75881295ff1ef P ac78ac69480a4042a6278f187e3b4d61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac78ac69480a4042a6278f187e3b4d61" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 38185 } } peers { permanent_uuid: "f6f4b69dba6341ff926f2501f7d96928" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 46057 } } peers { permanent_uuid: "063967dd04ee4fec97fccb5c6a5815f8" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 33411 } }
I20260812 02:31:51.294821 17226 consensus_queue.cc:260] T 16010c528ee0420b82f75881295ff1ef P 063967dd04ee4fec97fccb5c6a5815f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac78ac69480a4042a6278f187e3b4d61" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 38185 } } peers { permanent_uuid: "f6f4b69dba6341ff926f2501f7d96928" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 46057 } } peers { permanent_uuid: "063967dd04ee4fec97fccb5c6a5815f8" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 33411 } }
I20260812 02:31:51.295033 17225 ts_tablet_manager.cc:1428] T 16010c528ee0420b82f75881295ff1ef P ac78ac69480a4042a6278f187e3b4d61: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260812 02:31:51.295040 17227 tablet_bootstrap.cc:492] T 16010c528ee0420b82f75881295ff1ef P f6f4b69dba6341ff926f2501f7d96928: No bootstrap required, opened a new log
I20260812 02:31:51.295138 17227 ts_tablet_manager.cc:1397] T 16010c528ee0420b82f75881295ff1ef P f6f4b69dba6341ff926f2501f7d96928: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20260812 02:31:51.295199 17226 ts_tablet_manager.cc:1428] T 16010c528ee0420b82f75881295ff1ef P 063967dd04ee4fec97fccb5c6a5815f8: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260812 02:31:51.295570 17232 tablet_replica.cc:439] T 16010c528ee0420b82f75881295ff1ef P 063967dd04ee4fec97fccb5c6a5815f8: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac78ac69480a4042a6278f187e3b4d61" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 38185 } } peers { permanent_uuid: "f6f4b69dba6341ff926f2501f7d96928" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 46057 } } peers { permanent_uuid: "063967dd04ee4fec97fccb5c6a5815f8" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 33411 } } }
I20260812 02:31:51.295679 17227 raft_consensus.cc:348] T 16010c528ee0420b82f75881295ff1ef P f6f4b69dba6341ff926f2501f7d96928 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac78ac69480a4042a6278f187e3b4d61" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 38185 } } peers { permanent_uuid: "f6f4b69dba6341ff926f2501f7d96928" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 46057 } } peers { permanent_uuid: "063967dd04ee4fec97fccb5c6a5815f8" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 33411 } }
I20260812 02:31:51.295763 17232 tablet_replica.cc:442] T 16010c528ee0420b82f75881295ff1ef P 063967dd04ee4fec97fccb5c6a5815f8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 02:31:51.295805 17227 raft_consensus.cc:374] T 16010c528ee0420b82f75881295ff1ef P f6f4b69dba6341ff926f2501f7d96928 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:31:51.295849 17227 raft_consensus.cc:729] T 16010c528ee0420b82f75881295ff1ef P f6f4b69dba6341ff926f2501f7d96928 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6f4b69dba6341ff926f2501f7d96928, State: Initialized, Role: FOLLOWER
I20260812 02:31:51.295998 17227 consensus_queue.cc:260] T 16010c528ee0420b82f75881295ff1ef P f6f4b69dba6341ff926f2501f7d96928 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac78ac69480a4042a6278f187e3b4d61" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 38185 } } peers { permanent_uuid: "f6f4b69dba6341ff926f2501f7d96928" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 46057 } } peers { permanent_uuid: "063967dd04ee4fec97fccb5c6a5815f8" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 33411 } }
I20260812 02:31:51.296314 17227 ts_tablet_manager.cc:1428] T 16010c528ee0420b82f75881295ff1ef P f6f4b69dba6341ff926f2501f7d96928: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260812 02:31:51.297227 17231 tablet_replica.cc:439] T 16010c528ee0420b82f75881295ff1ef P ac78ac69480a4042a6278f187e3b4d61: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac78ac69480a4042a6278f187e3b4d61" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 38185 } } peers { permanent_uuid: "f6f4b69dba6341ff926f2501f7d96928" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 46057 } } peers { permanent_uuid: "063967dd04ee4fec97fccb5c6a5815f8" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 33411 } } }
I20260812 02:31:51.297443 17231 tablet_replica.cc:442] T 16010c528ee0420b82f75881295ff1ef P ac78ac69480a4042a6278f187e3b4d61: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 02:31:51.297878 17233 tablet_replica.cc:439] T 16010c528ee0420b82f75881295ff1ef P f6f4b69dba6341ff926f2501f7d96928: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac78ac69480a4042a6278f187e3b4d61" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 38185 } } peers { permanent_uuid: "f6f4b69dba6341ff926f2501f7d96928" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 46057 } } peers { permanent_uuid: "063967dd04ee4fec97fccb5c6a5815f8" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 33411 } } }
I20260812 02:31:51.298050 17233 tablet_replica.cc:442] T 16010c528ee0420b82f75881295ff1ef P f6f4b69dba6341ff926f2501f7d96928: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 02:31:51.340286 17232 raft_consensus.cc:482] T 16010c528ee0420b82f75881295ff1ef P 063967dd04ee4fec97fccb5c6a5815f8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 02:31:51.340492 17232 raft_consensus.cc:504] T 16010c528ee0420b82f75881295ff1ef P 063967dd04ee4fec97fccb5c6a5815f8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac78ac69480a4042a6278f187e3b4d61" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 38185 } } peers { permanent_uuid: "f6f4b69dba6341ff926f2501f7d96928" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 46057 } } peers { permanent_uuid: "063967dd04ee4fec97fccb5c6a5815f8" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 33411 } }
I20260812 02:31:51.342464 17232 leader_election.cc:288] T 16010c528ee0420b82f75881295ff1ef P 063967dd04ee4fec97fccb5c6a5815f8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac78ac69480a4042a6278f187e3b4d61 (127.14.103.1:38185), f6f4b69dba6341ff926f2501f7d96928 (127.14.103.2:46057)
I20260812 02:31:51.347832 17223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:31:51.342894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.3:35927 (local address 127.14.103.1:38185)
0812 02:31:51.343079 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0812 02:31:51.343084 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:31:51.343351 (+ 267us) server_negotiation.cc:407] Connection header received
0812 02:31:51.343560 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:31:51.343565 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:31:51.343621 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:31:51.343709 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:31:51.344445 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:31:51.345330 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:31:51.346527 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:31:51.347046 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:31:51.347212 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:31:51.347280 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0812 02:31:51.347593 (+ 313us) server_negotiation.cc:299] Negotiation successful
0812 02:31:51.347681 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260812 02:31:51.347857 17251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:31:51.343496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.3:59953 (local address 127.14.103.2:46057)
0812 02:31:51.343758 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0812 02:31:51.343777 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0812 02:31:51.343797 (+ 20us) server_negotiation.cc:407] Connection header received
0812 02:31:51.344138 (+ 341us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:31:51.344146 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:31:51.344208 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:31:51.344311 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:31:51.344673 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:31:51.345223 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:31:51.346649 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:31:51.347188 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:31:51.347350 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:31:51.347419 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0812 02:31:51.347592 (+ 173us) server_negotiation.cc:299] Negotiation successful
0812 02:31:51.347680 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":81,"threads_started":1}
I20260812 02:31:51.347869 17249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:31:51.342692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.1:38185 (local address 127.14.103.3:35927)
0812 02:31:51.343241 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0812 02:31:51.343261 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 02:31:51.343408 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:31:51.343745 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:31:51.343749 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:31:51.343775 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:31:51.344191 (+ 416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:31:51.344203 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:31:51.345477 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:31:51.345480 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:31:51.346366 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:31:51.346377 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:31:51.347225 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:31:51.347232 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:31:51.347410 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:31:51.347442 (+ 32us) client_negotiation.cc:769] Sending connection context
0812 02:31:51.347572 (+ 130us) client_negotiation.cc:240] Negotiation successful
0812 02:31:51.347683 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":349,"mutex_wait_us":95,"spinlock_wait_cycles":896,"thread_start_us":207,"threads_started":1}
I20260812 02:31:51.347867 17250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:31:51.343384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.2:46057 (local address 127.14.103.3:59953)
0812 02:31:51.343724 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0812 02:31:51.343737 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 02:31:51.343831 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:31:51.344419 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:31:51.344424 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:31:51.344445 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:31:51.344547 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:31:51.344552 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:31:51.345356 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:31:51.345360 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:31:51.346504 (+ 1144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:31:51.346516 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:31:51.347330 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:31:51.347333 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:31:51.347461 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:31:51.347479 (+ 18us) client_negotiation.cc:769] Sending connection context
0812 02:31:51.347573 (+ 94us) client_negotiation.cc:240] Negotiation successful
0812 02:31:51.347684 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"mutex_wait_us":38,"thread_start_us":77,"threads_started":1}
I20260812 02:31:51.348371 16836 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16010c528ee0420b82f75881295ff1ef" candidate_uuid: "063967dd04ee4fec97fccb5c6a5815f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac78ac69480a4042a6278f187e3b4d61" is_pre_election: true
I20260812 02:31:51.348578 16836 raft_consensus.cc:2454] T 16010c528ee0420b82f75881295ff1ef P ac78ac69480a4042a6278f187e3b4d61 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 063967dd04ee4fec97fccb5c6a5815f8 in term 0.
I20260812 02:31:51.348730 16989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16010c528ee0420b82f75881295ff1ef" candidate_uuid: "063967dd04ee4fec97fccb5c6a5815f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6f4b69dba6341ff926f2501f7d96928" is_pre_election: true
I20260812 02:31:51.348997 16989 raft_consensus.cc:2454] T 16010c528ee0420b82f75881295ff1ef P f6f4b69dba6341ff926f2501f7d96928 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 063967dd04ee4fec97fccb5c6a5815f8 in term 0.
I20260812 02:31:51.349020 17068 leader_election.cc:302] T 16010c528ee0420b82f75881295ff1ef P 063967dd04ee4fec97fccb5c6a5815f8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 063967dd04ee4fec97fccb5c6a5815f8, ac78ac69480a4042a6278f187e3b4d61; no voters:
I20260812 02:31:51.349414 17232 raft_consensus.cc:2781] T 16010c528ee0420b82f75881295ff1ef P 063967dd04ee4fec97fccb5c6a5815f8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 02:31:51.349501 17232 raft_consensus.cc:482] T 16010c528ee0420b82f75881295ff1ef P 063967dd04ee4fec97fccb5c6a5815f8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 02:31:51.349563 17232 raft_consensus.cc:3037] T 16010c528ee0420b82f75881295ff1ef P 063967dd04ee4fec97fccb5c6a5815f8 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:31:51.351188 17232 raft_consensus.cc:504] T 16010c528ee0420b82f75881295ff1ef P 063967dd04ee4fec97fccb5c6a5815f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac78ac69480a4042a6278f187e3b4d61" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 38185 } } peers { permanent_uuid: "f6f4b69dba6341ff926f2501f7d96928" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 46057 } } peers { permanent_uuid: "063967dd04ee4fec97fccb5c6a5815f8" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 33411 } }
I20260812 02:31:51.351621 17232 leader_election.cc:288] T 16010c528ee0420b82f75881295ff1ef P 063967dd04ee4fec97fccb5c6a5815f8 [CANDIDATE]: Term 1 election: Requested vote from peers ac78ac69480a4042a6278f187e3b4d61 (127.14.103.1:38185), f6f4b69dba6341ff926f2501f7d96928 (127.14.103.2:46057)
I20260812 02:31:51.352214 16836 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16010c528ee0420b82f75881295ff1ef" candidate_uuid: "063967dd04ee4fec97fccb5c6a5815f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac78ac69480a4042a6278f187e3b4d61"
I20260812 02:31:51.352334 16989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16010c528ee0420b82f75881295ff1ef" candidate_uuid: "063967dd04ee4fec97fccb5c6a5815f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6f4b69dba6341ff926f2501f7d96928"
I20260812 02:31:51.352437 16836 raft_consensus.cc:3037] T 16010c528ee0420b82f75881295ff1ef P ac78ac69480a4042a6278f187e3b4d61 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:31:51.352502 16989 raft_consensus.cc:3037] T 16010c528ee0420b82f75881295ff1ef P f6f4b69dba6341ff926f2501f7d96928 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:31:51.353587 16836 raft_consensus.cc:2454] T 16010c528ee0420b82f75881295ff1ef P ac78ac69480a4042a6278f187e3b4d61 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 063967dd04ee4fec97fccb5c6a5815f8 in term 1.
I20260812 02:31:51.353700 16989 raft_consensus.cc:2454] T 16010c528ee0420b82f75881295ff1ef P f6f4b69dba6341ff926f2501f7d96928 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 063967dd04ee4fec97fccb5c6a5815f8 in term 1.
I20260812 02:31:51.353981 17068 leader_election.cc:302] T 16010c528ee0420b82f75881295ff1ef P 063967dd04ee4fec97fccb5c6a5815f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 063967dd04ee4fec97fccb5c6a5815f8, ac78ac69480a4042a6278f187e3b4d61; no voters:
I20260812 02:31:51.354208 17232 raft_consensus.cc:2781] T 16010c528ee0420b82f75881295ff1ef P 063967dd04ee4fec97fccb5c6a5815f8 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:31:51.354450 17232 raft_consensus.cc:686] T 16010c528ee0420b82f75881295ff1ef P 063967dd04ee4fec97fccb5c6a5815f8 [term 1 LEADER]: Becoming Leader. State: Replica: 063967dd04ee4fec97fccb5c6a5815f8, State: Running, Role: LEADER
I20260812 02:31:51.354874 17232 consensus_queue.cc:237] T 16010c528ee0420b82f75881295ff1ef P 063967dd04ee4fec97fccb5c6a5815f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac78ac69480a4042a6278f187e3b4d61" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 38185 } } peers { permanent_uuid: "f6f4b69dba6341ff926f2501f7d96928" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 46057 } } peers { permanent_uuid: "063967dd04ee4fec97fccb5c6a5815f8" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 33411 } }
I20260812 02:31:51.357491 17252 tablet_replica.cc:439] T 16010c528ee0420b82f75881295ff1ef P 063967dd04ee4fec97fccb5c6a5815f8: TxnStatusTablet state changed. Reason: New leader 063967dd04ee4fec97fccb5c6a5815f8. Latest consensus state: current_term: 1 leader_uuid: "063967dd04ee4fec97fccb5c6a5815f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac78ac69480a4042a6278f187e3b4d61" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 38185 } } peers { permanent_uuid: "f6f4b69dba6341ff926f2501f7d96928" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 46057 } } peers { permanent_uuid: "063967dd04ee4fec97fccb5c6a5815f8" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 33411 } } }
I20260812 02:31:51.357754 17252 tablet_replica.cc:442] T 16010c528ee0420b82f75881295ff1ef P 063967dd04ee4fec97fccb5c6a5815f8: This TxnStatusTablet replica's current role is: LEADER
I20260812 02:31:51.358144 17254 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260812 02:31:51.358373 17254 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260812 02:31:51.359402 16659 catalog_manager.cc:5495] T 16010c528ee0420b82f75881295ff1ef P 063967dd04ee4fec97fccb5c6a5815f8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 063967dd04ee4fec97fccb5c6a5815f8 (127.14.103.3). New cstate: current_term: 1 leader_uuid: "063967dd04ee4fec97fccb5c6a5815f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac78ac69480a4042a6278f187e3b4d61" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 38185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f6f4b69dba6341ff926f2501f7d96928" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 46057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "063967dd04ee4fec97fccb5c6a5815f8" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 33411 } health_report { overall_health: HEALTHY } } }
I20260812 02:31:51.389807 17220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:31:51.380705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57530 (local address 127.14.103.3:33411)
0812 02:31:51.381231 (+ 526us) server_negotiation.cc:206] Beginning negotiation
0812 02:31:51.381237 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 02:31:51.381269 (+ 32us) server_negotiation.cc:407] Connection header received
0812 02:31:51.381592 (+ 323us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:31:51.381597 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:31:51.381685 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:31:51.382075 (+ 390us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 02:31:51.383656 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:31:51.385552 (+ 1896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:31:51.387260 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:31:51.387963 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:31:51.388295 (+ 332us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:31:51.388718 (+ 423us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 02:31:51.389270 (+ 552us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 02:31:51.389396 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0812 02:31:51.389559 (+ 163us) server_negotiation.cc:299] Negotiation successful
0812 02:31:51.389631 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278}
I20260812 02:31:51.389804 17245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:31:51.380546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.3:33411 (local address 127.0.0.1:57530)
0812 02:31:51.380992 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0812 02:31:51.381028 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0812 02:31:51.381177 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:31:51.382156 (+ 979us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:31:51.382159 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:31:51.382168 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 02:31:51.382286 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:31:51.382291 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:31:51.385990 (+ 3699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:31:51.385994 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:31:51.387052 (+ 1058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:31:51.387073 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:31:51.388285 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:31:51.388290 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:31:51.388409 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:31:51.388509 (+ 100us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 02:31:51.389409 (+ 900us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 02:31:51.389434 (+ 25us) client_negotiation.cc:769] Sending connection context
0812 02:31:51.389523 (+ 89us) client_negotiation.cc:240] Negotiation successful
0812 02:31:51.389644 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286}
I20260812 02:31:51.396330 16836 raft_consensus.cc:1261] T 16010c528ee0420b82f75881295ff1ef P ac78ac69480a4042a6278f187e3b4d61 [term 1 FOLLOWER]: Refusing update from remote peer 063967dd04ee4fec97fccb5c6a5815f8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:31:51.396329 16989 raft_consensus.cc:1261] T 16010c528ee0420b82f75881295ff1ef P f6f4b69dba6341ff926f2501f7d96928 [term 1 FOLLOWER]: Refusing update from remote peer 063967dd04ee4fec97fccb5c6a5815f8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:31:51.397724 17252 consensus_queue.cc:1035] T 16010c528ee0420b82f75881295ff1ef P 063967dd04ee4fec97fccb5c6a5815f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac78ac69480a4042a6278f187e3b4d61" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 38185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:31:51.398070 17232 consensus_queue.cc:1035] T 16010c528ee0420b82f75881295ff1ef P 063967dd04ee4fec97fccb5c6a5815f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6f4b69dba6341ff926f2501f7d96928" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 46057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:31:51.405170 17233 tablet_replica.cc:439] T 16010c528ee0420b82f75881295ff1ef P f6f4b69dba6341ff926f2501f7d96928: TxnStatusTablet state changed. Reason: New leader 063967dd04ee4fec97fccb5c6a5815f8. Latest consensus state: current_term: 1 leader_uuid: "063967dd04ee4fec97fccb5c6a5815f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac78ac69480a4042a6278f187e3b4d61" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 38185 } } peers { permanent_uuid: "f6f4b69dba6341ff926f2501f7d96928" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 46057 } } peers { permanent_uuid: "063967dd04ee4fec97fccb5c6a5815f8" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 33411 } } }
I20260812 02:31:51.405355 17233 tablet_replica.cc:442] T 16010c528ee0420b82f75881295ff1ef P f6f4b69dba6341ff926f2501f7d96928: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 02:31:51.405712 17231 tablet_replica.cc:439] T 16010c528ee0420b82f75881295ff1ef P ac78ac69480a4042a6278f187e3b4d61: TxnStatusTablet state changed. Reason: New leader 063967dd04ee4fec97fccb5c6a5815f8. Latest consensus state: current_term: 1 leader_uuid: "063967dd04ee4fec97fccb5c6a5815f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac78ac69480a4042a6278f187e3b4d61" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 38185 } } peers { permanent_uuid: "f6f4b69dba6341ff926f2501f7d96928" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 46057 } } peers { permanent_uuid: "063967dd04ee4fec97fccb5c6a5815f8" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 33411 } } }
I20260812 02:31:51.405867 17231 tablet_replica.cc:442] T 16010c528ee0420b82f75881295ff1ef P ac78ac69480a4042a6278f187e3b4d61: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 02:31:51.408363 17252 tablet_replica.cc:439] T 16010c528ee0420b82f75881295ff1ef P 063967dd04ee4fec97fccb5c6a5815f8: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "063967dd04ee4fec97fccb5c6a5815f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac78ac69480a4042a6278f187e3b4d61" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 38185 } } peers { permanent_uuid: "f6f4b69dba6341ff926f2501f7d96928" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 46057 } } peers { permanent_uuid: "063967dd04ee4fec97fccb5c6a5815f8" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 33411 } } }
I20260812 02:31:51.408528 17252 tablet_replica.cc:442] T 16010c528ee0420b82f75881295ff1ef P 063967dd04ee4fec97fccb5c6a5815f8: This TxnStatusTablet replica's current role is: LEADER
I20260812 02:31:51.408638 17252 tablet_replica.cc:439] T 16010c528ee0420b82f75881295ff1ef P 063967dd04ee4fec97fccb5c6a5815f8: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "063967dd04ee4fec97fccb5c6a5815f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac78ac69480a4042a6278f187e3b4d61" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 38185 } } peers { permanent_uuid: "f6f4b69dba6341ff926f2501f7d96928" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 46057 } } peers { permanent_uuid: "063967dd04ee4fec97fccb5c6a5815f8" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 33411 } } }
I20260812 02:31:51.408710 17252 tablet_replica.cc:442] T 16010c528ee0420b82f75881295ff1ef P 063967dd04ee4fec97fccb5c6a5815f8: This TxnStatusTablet replica's current role is: LEADER
I20260812 02:31:51.408639 17231 tablet_replica.cc:439] T 16010c528ee0420b82f75881295ff1ef P ac78ac69480a4042a6278f187e3b4d61: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "063967dd04ee4fec97fccb5c6a5815f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac78ac69480a4042a6278f187e3b4d61" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 38185 } } peers { permanent_uuid: "f6f4b69dba6341ff926f2501f7d96928" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 46057 } } peers { permanent_uuid: "063967dd04ee4fec97fccb5c6a5815f8" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 33411 } } }
I20260812 02:31:51.408778 17231 tablet_replica.cc:442] T 16010c528ee0420b82f75881295ff1ef P ac78ac69480a4042a6278f187e3b4d61: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 02:31:51.408819 17257 mvcc.cc:205] Tried to move back new op lower bound from 7317511829070995456 to 7317511828912877568. Current Snapshot: MvccSnapshot[applied={T|T < 7317511829070995456}]
I20260812 02:31:51.409699 17259 mvcc.cc:205] Tried to move back new op lower bound from 7317511829070995456 to 7317511828912877568. Current Snapshot: MvccSnapshot[applied={T|T < 7317511829070995456}]
I20260812 02:31:51.411891 17233 tablet_replica.cc:439] T 16010c528ee0420b82f75881295ff1ef P f6f4b69dba6341ff926f2501f7d96928: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "063967dd04ee4fec97fccb5c6a5815f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac78ac69480a4042a6278f187e3b4d61" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 38185 } } peers { permanent_uuid: "f6f4b69dba6341ff926f2501f7d96928" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 46057 } } peers { permanent_uuid: "063967dd04ee4fec97fccb5c6a5815f8" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 33411 } } }
I20260812 02:31:51.412072 17233 tablet_replica.cc:442] T 16010c528ee0420b82f75881295ff1ef P f6f4b69dba6341ff926f2501f7d96928: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 02:31:51.414076 17258 mvcc.cc:205] Tried to move back new op lower bound from 7317511829070995456 to 7317511828912877568. Current Snapshot: MvccSnapshot[applied={T|T < 7317511829070995456}]
I20260812 02:31:51.425287 17251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:31:51.420047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55188 (local address 127.14.103.2:46057)
0812 02:31:51.420225 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0812 02:31:51.420230 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:31:51.420301 (+ 71us) server_negotiation.cc:407] Connection header received
0812 02:31:51.420527 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:31:51.420531 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:31:51.420608 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:31:51.420729 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 02:31:51.421138 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:31:51.422016 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:31:51.423421 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:31:51.423667 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:31:51.423792 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:31:51.424119 (+ 327us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 02:31:51.424633 (+ 514us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 02:31:51.424772 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0812 02:31:51.424860 (+ 88us) server_negotiation.cc:299] Negotiation successful
0812 02:31:51.424930 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260812 02:31:51.440876 17220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:31:51.435813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.2:40433 (local address 127.14.103.3:33411)
0812 02:31:51.435958 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0812 02:31:51.435963 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:31:51.436177 (+ 214us) server_negotiation.cc:407] Connection header received
0812 02:31:51.436310 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:31:51.436313 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:31:51.436372 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:31:51.436477 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 02:31:51.436813 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:31:51.437759 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:31:51.438818 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:31:51.439630 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:31:51.439768 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:31:51.440283 (+ 515us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 02:31:51.440387 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 02:31:51.440470 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0812 02:31:51.440658 (+ 188us) server_negotiation.cc:299] Negotiation successful
0812 02:31:51.440727 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260812 02:31:51.440912 17270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:31:51.435695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.3:33411 (local address 127.14.103.2:40433)
0812 02:31:51.436083 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0812 02:31:51.436097 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 02:31:51.436214 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:31:51.436532 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:31:51.436535 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:31:51.436542 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 02:31:51.436671 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:31:51.436676 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:31:51.437955 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:31:51.437959 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:31:51.438663 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:31:51.438675 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:31:51.439819 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:31:51.439822 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:31:51.439936 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:31:51.440008 (+ 72us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 02:31:51.440533 (+ 525us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 02:31:51.440541 (+ 8us) client_negotiation.cc:769] Sending connection context
0812 02:31:51.440624 (+ 83us) client_negotiation.cc:240] Negotiation successful
0812 02:31:51.440718 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":139,"threads_started":1}
I20260812 02:31:51.450541 16836 raft_consensus.cc:1261] T 4440a6466c6643d0bfffee353db96a25 P ac78ac69480a4042a6278f187e3b4d61 [term 1 FOLLOWER]: Refusing update from remote peer f6f4b69dba6341ff926f2501f7d96928: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:31:51.450860 17142 raft_consensus.cc:1261] T 4440a6466c6643d0bfffee353db96a25 P 063967dd04ee4fec97fccb5c6a5815f8 [term 1 FOLLOWER]: Refusing update from remote peer f6f4b69dba6341ff926f2501f7d96928: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:31:51.451455 17237 consensus_queue.cc:1035] T 4440a6466c6643d0bfffee353db96a25 P f6f4b69dba6341ff926f2501f7d96928 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac78ac69480a4042a6278f187e3b4d61" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 38185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:31:51.451715 17233 consensus_queue.cc:1035] T 4440a6466c6643d0bfffee353db96a25 P f6f4b69dba6341ff926f2501f7d96928 [LEADER]: Connected to new peer: Peer: permanent_uuid: "063967dd04ee4fec97fccb5c6a5815f8" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 33411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:31:51.488081 17251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:31:51.482164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.3:36387 (local address 127.14.103.2:46057)
0812 02:31:51.482488 (+ 324us) server_negotiation.cc:206] Beginning negotiation
0812 02:31:51.482493 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:31:51.482782 (+ 289us) server_negotiation.cc:407] Connection header received
0812 02:31:51.482939 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:31:51.482943 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:31:51.483038 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:31:51.483481 (+ 443us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 02:31:51.484121 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:31:51.485196 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:31:51.486200 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:31:51.486565 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:31:51.486709 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:31:51.487447 (+ 738us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 02:31:51.487570 (+ 123us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 02:31:51.487650 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0812 02:31:51.487853 (+ 203us) server_negotiation.cc:299] Negotiation successful
0812 02:31:51.487919 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260812 02:31:51.488101 17277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:31:51.481754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.2:46057 (local address 127.14.103.3:36387)
0812 02:31:51.482674 (+ 920us) negotiation.cc:107] Waiting for socket to connect
0812 02:31:51.482692 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 02:31:51.482818 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:31:51.483483 (+ 665us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:31:51.483497 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:31:51.483519 (+ 22us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 02:31:51.483951 (+ 432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:31:51.483964 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:31:51.485349 (+ 1385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:31:51.485352 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:31:51.486019 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:31:51.486028 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:31:51.487037 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:31:51.487041 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:31:51.487174 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:31:51.487282 (+ 108us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 02:31:51.487710 (+ 428us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 02:31:51.487720 (+ 10us) client_negotiation.cc:769] Sending connection context
0812 02:31:51.487808 (+ 88us) client_negotiation.cc:240] Negotiation successful
0812 02:31:51.487929 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":761,"thread_start_us":159,"threads_started":1}
I20260812 02:31:51.520624 16659 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51182:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 02:31:51.520861 16659 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51182:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 02:31:51.525128 16659 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 43b9d054e04b419f9b79d092ff15ba6b: Sending DeleteTablet for 3 replicas of tablet 4440a6466c6643d0bfffee353db96a25
I20260812 02:31:51.526304 16814 tablet_service.cc:1514] Processing DeleteTablet for tablet 4440a6466c6643d0bfffee353db96a25 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:31:51 UTC) from {username='kudu', principal='kudu/127.14.103.62@KRBTEST.COM'} at 127.0.0.1:38290
I20260812 02:31:51.527298 17118 tablet_service.cc:1514] Processing DeleteTablet for tablet 4440a6466c6643d0bfffee353db96a25 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:31:51 UTC) from {username='kudu', principal='kudu/127.14.103.62@KRBTEST.COM'} at 127.0.0.1:57520
I20260812 02:31:51.527442 17280 tablet_replica.cc:330] T 4440a6466c6643d0bfffee353db96a25 P ac78ac69480a4042a6278f187e3b4d61: stopping tablet replica
I20260812 02:31:51.528007 16966 tablet_service.cc:1514] Processing DeleteTablet for tablet 4440a6466c6643d0bfffee353db96a25 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:31:51 UTC) from {username='kudu', principal='kudu/127.14.103.62@KRBTEST.COM'} at 127.0.0.1:55176
I20260812 02:31:51.528303 17280 raft_consensus.cc:2229] T 4440a6466c6643d0bfffee353db96a25 P ac78ac69480a4042a6278f187e3b4d61 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 02:31:51.528744 17280 raft_consensus.cc:2258] T 4440a6466c6643d0bfffee353db96a25 P ac78ac69480a4042a6278f187e3b4d61 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:31:51.529103 16659 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51182:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 02:31:51.529676 16659 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 02:31:51.531206 17284 tablet_replica.cc:330] T 4440a6466c6643d0bfffee353db96a25 P f6f4b69dba6341ff926f2501f7d96928: stopping tablet replica
I20260812 02:31:51.531621 17284 raft_consensus.cc:2229] T 4440a6466c6643d0bfffee353db96a25 P f6f4b69dba6341ff926f2501f7d96928 [term 1 LEADER]: Raft consensus shutting down.
I20260812 02:31:51.532086 17284 raft_consensus.cc:2258] T 4440a6466c6643d0bfffee353db96a25 P f6f4b69dba6341ff926f2501f7d96928 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:31:51.532619 17283 tablet_replica.cc:330] T 4440a6466c6643d0bfffee353db96a25 P 063967dd04ee4fec97fccb5c6a5815f8: stopping tablet replica
I20260812 02:31:51.533008 17283 raft_consensus.cc:2229] T 4440a6466c6643d0bfffee353db96a25 P 063967dd04ee4fec97fccb5c6a5815f8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 02:31:51.533202 17283 raft_consensus.cc:2258] T 4440a6466c6643d0bfffee353db96a25 P 063967dd04ee4fec97fccb5c6a5815f8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:31:51.533727 17284 ts_tablet_manager.cc:1905] T 4440a6466c6643d0bfffee353db96a25 P f6f4b69dba6341ff926f2501f7d96928: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:31:51.534678 17283 ts_tablet_manager.cc:1905] T 4440a6466c6643d0bfffee353db96a25 P 063967dd04ee4fec97fccb5c6a5815f8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:31:51.545269 17283 ts_tablet_manager.cc:1918] T 4440a6466c6643d0bfffee353db96a25 P 063967dd04ee4fec97fccb5c6a5815f8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 02:31:51.545501 17283 log.cc:1196] T 4440a6466c6643d0bfffee353db96a25 P 063967dd04ee4fec97fccb5c6a5815f8: Deleting WAL directory at /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501898713699-14748-0/minicluster-data/ts-2/wal/wals/4440a6466c6643d0bfffee353db96a25
I20260812 02:31:51.546000 17283 ts_tablet_manager.cc:1939] T 4440a6466c6643d0bfffee353db96a25 P 063967dd04ee4fec97fccb5c6a5815f8: Deleting consensus metadata
I20260812 02:31:51.546386 16814 tablet_service.cc:1467] Processing CreateTablet for tablet a4a37b9b3a504a88beabf5a0877fbf7e (DEFAULT_TABLE table=test-table [id=88e73b2e1a3f4912899f5c2191045471]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:31:51.546720 16814 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4a37b9b3a504a88beabf5a0877fbf7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:31:51.547901 16647 catalog_manager.cc:4841] TS 063967dd04ee4fec97fccb5c6a5815f8 (127.14.103.3:33411): tablet 4440a6466c6643d0bfffee353db96a25 (table test-table [id=1fb79426247e406eb08e908e72658b2d]) successfully deleted
I20260812 02:31:51.548548 17118 tablet_service.cc:1467] Processing CreateTablet for tablet a4a37b9b3a504a88beabf5a0877fbf7e (DEFAULT_TABLE table=test-table [id=88e73b2e1a3f4912899f5c2191045471]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:31:51.548906 17118 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4a37b9b3a504a88beabf5a0877fbf7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:31:51.549958 17225 tablet_bootstrap.cc:492] T a4a37b9b3a504a88beabf5a0877fbf7e P ac78ac69480a4042a6278f187e3b4d61: Bootstrap starting.
I20260812 02:31:51.551188 17225 tablet_bootstrap.cc:654] T a4a37b9b3a504a88beabf5a0877fbf7e P ac78ac69480a4042a6278f187e3b4d61: Neither blocks nor log segments found. Creating new log.
I20260812 02:31:51.552274 17226 tablet_bootstrap.cc:492] T a4a37b9b3a504a88beabf5a0877fbf7e P 063967dd04ee4fec97fccb5c6a5815f8: Bootstrap starting.
I20260812 02:31:51.552731 16966 tablet_service.cc:1467] Processing CreateTablet for tablet a4a37b9b3a504a88beabf5a0877fbf7e (DEFAULT_TABLE table=test-table [id=88e73b2e1a3f4912899f5c2191045471]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:31:51.553148 16966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4a37b9b3a504a88beabf5a0877fbf7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:31:51.553321 17225 tablet_bootstrap.cc:492] T a4a37b9b3a504a88beabf5a0877fbf7e P ac78ac69480a4042a6278f187e3b4d61: No bootstrap required, opened a new log
I20260812 02:31:51.553431 17225 ts_tablet_manager.cc:1397] T a4a37b9b3a504a88beabf5a0877fbf7e P ac78ac69480a4042a6278f187e3b4d61: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20260812 02:31:51.553522 17226 tablet_bootstrap.cc:654] T a4a37b9b3a504a88beabf5a0877fbf7e P 063967dd04ee4fec97fccb5c6a5815f8: Neither blocks nor log segments found. Creating new log.
I20260812 02:31:51.554030 17225 raft_consensus.cc:348] T a4a37b9b3a504a88beabf5a0877fbf7e P ac78ac69480a4042a6278f187e3b4d61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac78ac69480a4042a6278f187e3b4d61" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 38185 } } peers { permanent_uuid: "f6f4b69dba6341ff926f2501f7d96928" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 46057 } } peers { permanent_uuid: "063967dd04ee4fec97fccb5c6a5815f8" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 33411 } }
I20260812 02:31:51.554174 17225 raft_consensus.cc:374] T a4a37b9b3a504a88beabf5a0877fbf7e P ac78ac69480a4042a6278f187e3b4d61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:31:51.554208 17225 raft_consensus.cc:729] T a4a37b9b3a504a88beabf5a0877fbf7e P ac78ac69480a4042a6278f187e3b4d61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac78ac69480a4042a6278f187e3b4d61, State: Initialized, Role: FOLLOWER
I20260812 02:31:51.554338 17225 consensus_queue.cc:260] T a4a37b9b3a504a88beabf5a0877fbf7e P ac78ac69480a4042a6278f187e3b4d61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac78ac69480a4042a6278f187e3b4d61" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 38185 } } peers { permanent_uuid: "f6f4b69dba6341ff926f2501f7d96928" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 46057 } } peers { permanent_uuid: "063967dd04ee4fec97fccb5c6a5815f8" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 33411 } }
I20260812 02:31:51.554652 17225 ts_tablet_manager.cc:1428] T a4a37b9b3a504a88beabf5a0877fbf7e P ac78ac69480a4042a6278f187e3b4d61: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260812 02:31:51.542176 17284 ts_tablet_manager.cc:1918] T 4440a6466c6643d0bfffee353db96a25 P f6f4b69dba6341ff926f2501f7d96928: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 02:31:51.555284 17284 log.cc:1196] T 4440a6466c6643d0bfffee353db96a25 P f6f4b69dba6341ff926f2501f7d96928: Deleting WAL directory at /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501898713699-14748-0/minicluster-data/ts-1/wal/wals/4440a6466c6643d0bfffee353db96a25
I20260812 02:31:51.555670 17284 ts_tablet_manager.cc:1939] T 4440a6466c6643d0bfffee353db96a25 P f6f4b69dba6341ff926f2501f7d96928: Deleting consensus metadata
I20260812 02:31:51.557960 17227 tablet_bootstrap.cc:492] T a4a37b9b3a504a88beabf5a0877fbf7e P f6f4b69dba6341ff926f2501f7d96928: Bootstrap starting.
I20260812 02:31:51.558580 17280 ts_tablet_manager.cc:1905] T 4440a6466c6643d0bfffee353db96a25 P ac78ac69480a4042a6278f187e3b4d61: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:31:51.559173 17227 tablet_bootstrap.cc:654] T a4a37b9b3a504a88beabf5a0877fbf7e P f6f4b69dba6341ff926f2501f7d96928: Neither blocks nor log segments found. Creating new log.
I20260812 02:31:51.560338 16649 catalog_manager.cc:4841] TS f6f4b69dba6341ff926f2501f7d96928 (127.14.103.2:46057): tablet 4440a6466c6643d0bfffee353db96a25 (table test-table [id=1fb79426247e406eb08e908e72658b2d]) successfully deleted
I20260812 02:31:51.562579 17280 ts_tablet_manager.cc:1918] T 4440a6466c6643d0bfffee353db96a25 P ac78ac69480a4042a6278f187e3b4d61: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 02:31:51.562716 17280 log.cc:1196] T 4440a6466c6643d0bfffee353db96a25 P ac78ac69480a4042a6278f187e3b4d61: Deleting WAL directory at /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501898713699-14748-0/minicluster-data/ts-0/wal/wals/4440a6466c6643d0bfffee353db96a25
I20260812 02:31:51.563275 17280 ts_tablet_manager.cc:1939] T 4440a6466c6643d0bfffee353db96a25 P ac78ac69480a4042a6278f187e3b4d61: Deleting consensus metadata
I20260812 02:31:51.565689 17227 tablet_bootstrap.cc:492] T a4a37b9b3a504a88beabf5a0877fbf7e P f6f4b69dba6341ff926f2501f7d96928: No bootstrap required, opened a new log
I20260812 02:31:51.565876 17227 ts_tablet_manager.cc:1397] T a4a37b9b3a504a88beabf5a0877fbf7e P f6f4b69dba6341ff926f2501f7d96928: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.002s
I20260812 02:31:51.565934 17226 tablet_bootstrap.cc:492] T a4a37b9b3a504a88beabf5a0877fbf7e P 063967dd04ee4fec97fccb5c6a5815f8: No bootstrap required, opened a new log
I20260812 02:31:51.566025 17226 ts_tablet_manager.cc:1397] T a4a37b9b3a504a88beabf5a0877fbf7e P 063967dd04ee4fec97fccb5c6a5815f8: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.002s
I20260812 02:31:51.566454 17227 raft_consensus.cc:348] T a4a37b9b3a504a88beabf5a0877fbf7e P f6f4b69dba6341ff926f2501f7d96928 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac78ac69480a4042a6278f187e3b4d61" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 38185 } } peers { permanent_uuid: "f6f4b69dba6341ff926f2501f7d96928" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 46057 } } peers { permanent_uuid: "063967dd04ee4fec97fccb5c6a5815f8" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 33411 } }
I20260812 02:31:51.566468 17226 raft_consensus.cc:348] T a4a37b9b3a504a88beabf5a0877fbf7e P 063967dd04ee4fec97fccb5c6a5815f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac78ac69480a4042a6278f187e3b4d61" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 38185 } } peers { permanent_uuid: "f6f4b69dba6341ff926f2501f7d96928" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 46057 } } peers { permanent_uuid: "063967dd04ee4fec97fccb5c6a5815f8" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 33411 } }
I20260812 02:31:51.566586 17227 raft_consensus.cc:374] T a4a37b9b3a504a88beabf5a0877fbf7e P f6f4b69dba6341ff926f2501f7d96928 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:31:51.566586 17226 raft_consensus.cc:374] T a4a37b9b3a504a88beabf5a0877fbf7e P 063967dd04ee4fec97fccb5c6a5815f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:31:51.566620 17227 raft_consensus.cc:729] T a4a37b9b3a504a88beabf5a0877fbf7e P f6f4b69dba6341ff926f2501f7d96928 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6f4b69dba6341ff926f2501f7d96928, State: Initialized, Role: FOLLOWER
I20260812 02:31:51.566620 17226 raft_consensus.cc:729] T a4a37b9b3a504a88beabf5a0877fbf7e P 063967dd04ee4fec97fccb5c6a5815f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 063967dd04ee4fec97fccb5c6a5815f8, State: Initialized, Role: FOLLOWER
I20260812 02:31:51.566771 17227 consensus_queue.cc:260] T a4a37b9b3a504a88beabf5a0877fbf7e P f6f4b69dba6341ff926f2501f7d96928 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac78ac69480a4042a6278f187e3b4d61" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 38185 } } peers { permanent_uuid: "f6f4b69dba6341ff926f2501f7d96928" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 46057 } } peers { permanent_uuid: "063967dd04ee4fec97fccb5c6a5815f8" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 33411 } }
I20260812 02:31:51.566771 17226 consensus_queue.cc:260] T a4a37b9b3a504a88beabf5a0877fbf7e P 063967dd04ee4fec97fccb5c6a5815f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac78ac69480a4042a6278f187e3b4d61" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 38185 } } peers { permanent_uuid: "f6f4b69dba6341ff926f2501f7d96928" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 46057 } } peers { permanent_uuid: "063967dd04ee4fec97fccb5c6a5815f8" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 33411 } }
I20260812 02:31:51.567098 17227 ts_tablet_manager.cc:1428] T a4a37b9b3a504a88beabf5a0877fbf7e P f6f4b69dba6341ff926f2501f7d96928: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260812 02:31:51.567098 17226 ts_tablet_manager.cc:1428] T a4a37b9b3a504a88beabf5a0877fbf7e P 063967dd04ee4fec97fccb5c6a5815f8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260812 02:31:51.568503 16649 catalog_manager.cc:4841] TS ac78ac69480a4042a6278f187e3b4d61 (127.14.103.1:38185): tablet 4440a6466c6643d0bfffee353db96a25 (table test-table [id=1fb79426247e406eb08e908e72658b2d]) successfully deleted
I20260812 02:31:51.719707 17273 raft_consensus.cc:482] T a4a37b9b3a504a88beabf5a0877fbf7e P f6f4b69dba6341ff926f2501f7d96928 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 02:31:51.719907 17273 raft_consensus.cc:504] T a4a37b9b3a504a88beabf5a0877fbf7e P f6f4b69dba6341ff926f2501f7d96928 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac78ac69480a4042a6278f187e3b4d61" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 38185 } } peers { permanent_uuid: "f6f4b69dba6341ff926f2501f7d96928" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 46057 } } peers { permanent_uuid: "063967dd04ee4fec97fccb5c6a5815f8" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 33411 } }
I20260812 02:31:51.720506 17273 leader_election.cc:288] T a4a37b9b3a504a88beabf5a0877fbf7e P f6f4b69dba6341ff926f2501f7d96928 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac78ac69480a4042a6278f187e3b4d61 (127.14.103.1:38185), 063967dd04ee4fec97fccb5c6a5815f8 (127.14.103.3:33411)
I20260812 02:31:51.721024 16836 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4a37b9b3a504a88beabf5a0877fbf7e" candidate_uuid: "f6f4b69dba6341ff926f2501f7d96928" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac78ac69480a4042a6278f187e3b4d61" is_pre_election: true
I20260812 02:31:51.721040 17142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4a37b9b3a504a88beabf5a0877fbf7e" candidate_uuid: "f6f4b69dba6341ff926f2501f7d96928" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "063967dd04ee4fec97fccb5c6a5815f8" is_pre_election: true
I20260812 02:31:51.721311 17142 raft_consensus.cc:2454] T a4a37b9b3a504a88beabf5a0877fbf7e P 063967dd04ee4fec97fccb5c6a5815f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6f4b69dba6341ff926f2501f7d96928 in term 0.
I20260812 02:31:51.721306 16836 raft_consensus.cc:2454] T a4a37b9b3a504a88beabf5a0877fbf7e P ac78ac69480a4042a6278f187e3b4d61 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6f4b69dba6341ff926f2501f7d96928 in term 0.
I20260812 02:31:51.721728 16916 leader_election.cc:302] T a4a37b9b3a504a88beabf5a0877fbf7e P f6f4b69dba6341ff926f2501f7d96928 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac78ac69480a4042a6278f187e3b4d61, f6f4b69dba6341ff926f2501f7d96928; no voters:
I20260812 02:31:51.722002 17273 raft_consensus.cc:2781] T a4a37b9b3a504a88beabf5a0877fbf7e P f6f4b69dba6341ff926f2501f7d96928 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 02:31:51.722069 17273 raft_consensus.cc:482] T a4a37b9b3a504a88beabf5a0877fbf7e P f6f4b69dba6341ff926f2501f7d96928 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 02:31:51.722235 17273 raft_consensus.cc:3037] T a4a37b9b3a504a88beabf5a0877fbf7e P f6f4b69dba6341ff926f2501f7d96928 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:31:51.723798 17273 raft_consensus.cc:504] T a4a37b9b3a504a88beabf5a0877fbf7e P f6f4b69dba6341ff926f2501f7d96928 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac78ac69480a4042a6278f187e3b4d61" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 38185 } } peers { permanent_uuid: "f6f4b69dba6341ff926f2501f7d96928" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 46057 } } peers { permanent_uuid: "063967dd04ee4fec97fccb5c6a5815f8" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 33411 } }
I20260812 02:31:51.724493 17273 leader_election.cc:288] T a4a37b9b3a504a88beabf5a0877fbf7e P f6f4b69dba6341ff926f2501f7d96928 [CANDIDATE]: Term 1 election: Requested vote from peers ac78ac69480a4042a6278f187e3b4d61 (127.14.103.1:38185), 063967dd04ee4fec97fccb5c6a5815f8 (127.14.103.3:33411)
I20260812 02:31:51.725046 17142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4a37b9b3a504a88beabf5a0877fbf7e" candidate_uuid: "f6f4b69dba6341ff926f2501f7d96928" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "063967dd04ee4fec97fccb5c6a5815f8"
I20260812 02:31:51.725070 16836 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4a37b9b3a504a88beabf5a0877fbf7e" candidate_uuid: "f6f4b69dba6341ff926f2501f7d96928" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac78ac69480a4042a6278f187e3b4d61"
I20260812 02:31:51.725225 16836 raft_consensus.cc:3037] T a4a37b9b3a504a88beabf5a0877fbf7e P ac78ac69480a4042a6278f187e3b4d61 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:31:51.725226 17142 raft_consensus.cc:3037] T a4a37b9b3a504a88beabf5a0877fbf7e P 063967dd04ee4fec97fccb5c6a5815f8 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:31:51.726514 17142 raft_consensus.cc:2454] T a4a37b9b3a504a88beabf5a0877fbf7e P 063967dd04ee4fec97fccb5c6a5815f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6f4b69dba6341ff926f2501f7d96928 in term 1.
I20260812 02:31:51.726541 16836 raft_consensus.cc:2454] T a4a37b9b3a504a88beabf5a0877fbf7e P ac78ac69480a4042a6278f187e3b4d61 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6f4b69dba6341ff926f2501f7d96928 in term 1.
I20260812 02:31:51.726990 16914 leader_election.cc:302] T a4a37b9b3a504a88beabf5a0877fbf7e P f6f4b69dba6341ff926f2501f7d96928 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 063967dd04ee4fec97fccb5c6a5815f8, f6f4b69dba6341ff926f2501f7d96928; no voters:
I20260812 02:31:51.727231 17273 raft_consensus.cc:2781] T a4a37b9b3a504a88beabf5a0877fbf7e P f6f4b69dba6341ff926f2501f7d96928 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:31:51.727360 17273 raft_consensus.cc:686] T a4a37b9b3a504a88beabf5a0877fbf7e P f6f4b69dba6341ff926f2501f7d96928 [term 1 LEADER]: Becoming Leader. State: Replica: f6f4b69dba6341ff926f2501f7d96928, State: Running, Role: LEADER
I20260812 02:31:51.727550 17273 consensus_queue.cc:237] T a4a37b9b3a504a88beabf5a0877fbf7e P f6f4b69dba6341ff926f2501f7d96928 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac78ac69480a4042a6278f187e3b4d61" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 38185 } } peers { permanent_uuid: "f6f4b69dba6341ff926f2501f7d96928" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 46057 } } peers { permanent_uuid: "063967dd04ee4fec97fccb5c6a5815f8" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 33411 } }
I20260812 02:31:51.730090 16664 catalog_manager.cc:5495] T a4a37b9b3a504a88beabf5a0877fbf7e P f6f4b69dba6341ff926f2501f7d96928 reported cstate change: term changed from 0 to 1, leader changed from <none> to f6f4b69dba6341ff926f2501f7d96928 (127.14.103.2). New cstate: current_term: 1 leader_uuid: "f6f4b69dba6341ff926f2501f7d96928" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac78ac69480a4042a6278f187e3b4d61" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 38185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f6f4b69dba6341ff926f2501f7d96928" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 46057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "063967dd04ee4fec97fccb5c6a5815f8" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 33411 } health_report { overall_health: UNKNOWN } } }
I20260812 02:31:51.789824 17142 raft_consensus.cc:1261] T a4a37b9b3a504a88beabf5a0877fbf7e P 063967dd04ee4fec97fccb5c6a5815f8 [term 1 FOLLOWER]: Refusing update from remote peer f6f4b69dba6341ff926f2501f7d96928: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:31:51.790028 16836 raft_consensus.cc:1261] T a4a37b9b3a504a88beabf5a0877fbf7e P ac78ac69480a4042a6278f187e3b4d61 [term 1 FOLLOWER]: Refusing update from remote peer f6f4b69dba6341ff926f2501f7d96928: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:31:51.790400 17273 consensus_queue.cc:1035] T a4a37b9b3a504a88beabf5a0877fbf7e P f6f4b69dba6341ff926f2501f7d96928 [LEADER]: Connected to new peer: Peer: permanent_uuid: "063967dd04ee4fec97fccb5c6a5815f8" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 33411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:31:51.790568 17274 consensus_queue.cc:1035] T a4a37b9b3a504a88beabf5a0877fbf7e P f6f4b69dba6341ff926f2501f7d96928 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac78ac69480a4042a6278f187e3b4d61" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 38185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260812 02:31:51.809894 17258 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260812 02:31:53.385027 16663 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51182:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 02:31:53.385213 16663 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51182:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 02:31:53.388649 16663 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 43b9d054e04b419f9b79d092ff15ba6b: Sending DeleteTablet for 3 replicas of tablet a4a37b9b3a504a88beabf5a0877fbf7e
I20260812 02:31:53.389448 17118 tablet_service.cc:1514] Processing DeleteTablet for tablet a4a37b9b3a504a88beabf5a0877fbf7e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:31:53 UTC) from {username='kudu', principal='kudu/127.14.103.62@KRBTEST.COM'} at 127.0.0.1:57520
I20260812 02:31:53.389551 16814 tablet_service.cc:1514] Processing DeleteTablet for tablet a4a37b9b3a504a88beabf5a0877fbf7e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:31:53 UTC) from {username='kudu', principal='kudu/127.14.103.62@KRBTEST.COM'} at 127.0.0.1:38290
I20260812 02:31:53.389923 17314 tablet_replica.cc:330] T a4a37b9b3a504a88beabf5a0877fbf7e P 063967dd04ee4fec97fccb5c6a5815f8: stopping tablet replica
I20260812 02:31:53.390065 16966 tablet_service.cc:1514] Processing DeleteTablet for tablet a4a37b9b3a504a88beabf5a0877fbf7e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:31:53 UTC) from {username='kudu', principal='kudu/127.14.103.62@KRBTEST.COM'} at 127.0.0.1:55176
I20260812 02:31:53.390436 17316 tablet_replica.cc:330] T a4a37b9b3a504a88beabf5a0877fbf7e P f6f4b69dba6341ff926f2501f7d96928: stopping tablet replica
I20260812 02:31:53.390434 17314 raft_consensus.cc:2229] T a4a37b9b3a504a88beabf5a0877fbf7e P 063967dd04ee4fec97fccb5c6a5815f8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 02:31:53.390583 17314 raft_consensus.cc:2258] T a4a37b9b3a504a88beabf5a0877fbf7e P 063967dd04ee4fec97fccb5c6a5815f8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:31:53.390683 17316 raft_consensus.cc:2229] T a4a37b9b3a504a88beabf5a0877fbf7e P f6f4b69dba6341ff926f2501f7d96928 [term 1 LEADER]: Raft consensus shutting down.
I20260812 02:31:53.390839 17315 tablet_replica.cc:330] T a4a37b9b3a504a88beabf5a0877fbf7e P ac78ac69480a4042a6278f187e3b4d61: stopping tablet replica
I20260812 02:31:53.391003 17315 raft_consensus.cc:2229] T a4a37b9b3a504a88beabf5a0877fbf7e P ac78ac69480a4042a6278f187e3b4d61 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 02:31:53.391029 17316 raft_consensus.cc:2258] T a4a37b9b3a504a88beabf5a0877fbf7e P f6f4b69dba6341ff926f2501f7d96928 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:31:53.391139 17315 raft_consensus.cc:2258] T a4a37b9b3a504a88beabf5a0877fbf7e P ac78ac69480a4042a6278f187e3b4d61 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:31:53.391810 17316 ts_tablet_manager.cc:1905] T a4a37b9b3a504a88beabf5a0877fbf7e P f6f4b69dba6341ff926f2501f7d96928: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:31:53.391810 17315 ts_tablet_manager.cc:1905] T a4a37b9b3a504a88beabf5a0877fbf7e P ac78ac69480a4042a6278f187e3b4d61: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:31:53.395140 17315 ts_tablet_manager.cc:1918] T a4a37b9b3a504a88beabf5a0877fbf7e P ac78ac69480a4042a6278f187e3b4d61: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260812 02:31:53.395259 17315 log.cc:1196] T a4a37b9b3a504a88beabf5a0877fbf7e P ac78ac69480a4042a6278f187e3b4d61: Deleting WAL directory at /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501898713699-14748-0/minicluster-data/ts-0/wal/wals/a4a37b9b3a504a88beabf5a0877fbf7e
I20260812 02:31:53.395186 17316 ts_tablet_manager.cc:1918] T a4a37b9b3a504a88beabf5a0877fbf7e P f6f4b69dba6341ff926f2501f7d96928: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260812 02:31:53.395448 17316 log.cc:1196] T a4a37b9b3a504a88beabf5a0877fbf7e P f6f4b69dba6341ff926f2501f7d96928: Deleting WAL directory at /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501898713699-14748-0/minicluster-data/ts-1/wal/wals/a4a37b9b3a504a88beabf5a0877fbf7e
I20260812 02:31:53.395555 17314 ts_tablet_manager.cc:1905] T a4a37b9b3a504a88beabf5a0877fbf7e P 063967dd04ee4fec97fccb5c6a5815f8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:31:53.395695 17315 ts_tablet_manager.cc:1939] T a4a37b9b3a504a88beabf5a0877fbf7e P ac78ac69480a4042a6278f187e3b4d61: Deleting consensus metadata
I20260812 02:31:53.395947 17316 ts_tablet_manager.cc:1939] T a4a37b9b3a504a88beabf5a0877fbf7e P f6f4b69dba6341ff926f2501f7d96928: Deleting consensus metadata
I20260812 02:31:53.396615 16649 catalog_manager.cc:4841] TS ac78ac69480a4042a6278f187e3b4d61 (127.14.103.1:38185): tablet a4a37b9b3a504a88beabf5a0877fbf7e (table test-table [id=88e73b2e1a3f4912899f5c2191045471]) successfully deleted
I20260812 02:31:53.396855 16649 catalog_manager.cc:4841] TS f6f4b69dba6341ff926f2501f7d96928 (127.14.103.2:46057): tablet a4a37b9b3a504a88beabf5a0877fbf7e (table test-table [id=88e73b2e1a3f4912899f5c2191045471]) successfully deleted
I20260812 02:31:53.399245 17314 ts_tablet_manager.cc:1918] T a4a37b9b3a504a88beabf5a0877fbf7e P 063967dd04ee4fec97fccb5c6a5815f8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260812 02:31:53.399372 17314 log.cc:1196] T a4a37b9b3a504a88beabf5a0877fbf7e P 063967dd04ee4fec97fccb5c6a5815f8: Deleting WAL directory at /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501898713699-14748-0/minicluster-data/ts-2/wal/wals/a4a37b9b3a504a88beabf5a0877fbf7e
I20260812 02:31:53.399649 17314 ts_tablet_manager.cc:1939] T a4a37b9b3a504a88beabf5a0877fbf7e P 063967dd04ee4fec97fccb5c6a5815f8: Deleting consensus metadata
I20260812 02:31:53.402592 16647 catalog_manager.cc:4841] TS 063967dd04ee4fec97fccb5c6a5815f8 (127.14.103.3:33411): tablet a4a37b9b3a504a88beabf5a0877fbf7e (table test-table [id=88e73b2e1a3f4912899f5c2191045471]) successfully deleted
Aug 12 02:31:53 dist-test-slave-2w3w krb5kdc[16605](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501913, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 12 02:31:53 dist-test-slave-2w3w krb5kdc[16605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501913, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.14.103.62@KRBTEST.COM
I20260812 02:31:53.443605 17326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:31:53.424654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51194 (local address 127.14.103.62:43757)
0812 02:31:53.425547 (+ 893us) server_negotiation.cc:206] Beginning negotiation
0812 02:31:53.425562 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0812 02:31:53.425683 (+ 121us) server_negotiation.cc:407] Connection header received
0812 02:31:53.426013 (+ 330us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:31:53.426016 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:31:53.426071 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:31:53.426481 (+ 410us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:31:53.427550 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:31:53.428188 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:31:53.429105 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:31:53.429358 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:31:53.429477 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:31:53.439416 (+ 9939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:31:53.439421 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:31:53.439423 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:31:53.439461 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:31:53.441481 (+ 2020us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:31:53.442041 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:31:53.442044 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:31:53.442046 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:31:53.442247 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:31:53.442569 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:31:53.442572 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:31:53.442573 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:31:53.442787 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:31:53.442918 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0812 02:31:53.443296 (+ 378us) server_negotiation.cc:299] Negotiation successful
0812 02:31:53.443455 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":120,"threads_started":1}
I20260812 02:31:53.447065 16663 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51194:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 02:31:53.447626 16663 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 02:31:53.453524 17118 tablet_service.cc:1467] Processing CreateTablet for tablet a5586833c799404ca735aea68923b550 (DEFAULT_TABLE table=test-table [id=f9f0c51b99334a02803efecf69e837dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:31:53.453510 16966 tablet_service.cc:1467] Processing CreateTablet for tablet a5586833c799404ca735aea68923b550 (DEFAULT_TABLE table=test-table [id=f9f0c51b99334a02803efecf69e837dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:31:53.453510 16814 tablet_service.cc:1467] Processing CreateTablet for tablet a5586833c799404ca735aea68923b550 (DEFAULT_TABLE table=test-table [id=f9f0c51b99334a02803efecf69e837dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:31:53.453871 16814 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5586833c799404ca735aea68923b550. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:31:53.453881 17118 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5586833c799404ca735aea68923b550. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:31:53.453871 16966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5586833c799404ca735aea68923b550. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:31:53.456893 17328 tablet_bootstrap.cc:492] T a5586833c799404ca735aea68923b550 P f6f4b69dba6341ff926f2501f7d96928: Bootstrap starting.
I20260812 02:31:53.457392 17329 tablet_bootstrap.cc:492] T a5586833c799404ca735aea68923b550 P 063967dd04ee4fec97fccb5c6a5815f8: Bootstrap starting.
I20260812 02:31:53.458844 17329 tablet_bootstrap.cc:654] T a5586833c799404ca735aea68923b550 P 063967dd04ee4fec97fccb5c6a5815f8: Neither blocks nor log segments found. Creating new log.
I20260812 02:31:53.458865 17328 tablet_bootstrap.cc:654] T a5586833c799404ca735aea68923b550 P f6f4b69dba6341ff926f2501f7d96928: Neither blocks nor log segments found. Creating new log.
I20260812 02:31:53.459847 17327 tablet_bootstrap.cc:492] T a5586833c799404ca735aea68923b550 P ac78ac69480a4042a6278f187e3b4d61: Bootstrap starting.
I20260812 02:31:53.460624 17328 tablet_bootstrap.cc:492] T a5586833c799404ca735aea68923b550 P f6f4b69dba6341ff926f2501f7d96928: No bootstrap required, opened a new log
I20260812 02:31:53.460697 17329 tablet_bootstrap.cc:492] T a5586833c799404ca735aea68923b550 P 063967dd04ee4fec97fccb5c6a5815f8: No bootstrap required, opened a new log
I20260812 02:31:53.460865 17328 ts_tablet_manager.cc:1397] T a5586833c799404ca735aea68923b550 P f6f4b69dba6341ff926f2501f7d96928: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260812 02:31:53.460887 17329 ts_tablet_manager.cc:1397] T a5586833c799404ca735aea68923b550 P 063967dd04ee4fec97fccb5c6a5815f8: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260812 02:31:53.461081 17327 tablet_bootstrap.cc:654] T a5586833c799404ca735aea68923b550 P ac78ac69480a4042a6278f187e3b4d61: Neither blocks nor log segments found. Creating new log.
I20260812 02:31:53.461439 17329 raft_consensus.cc:348] T a5586833c799404ca735aea68923b550 P 063967dd04ee4fec97fccb5c6a5815f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac78ac69480a4042a6278f187e3b4d61" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 38185 } } peers { permanent_uuid: "f6f4b69dba6341ff926f2501f7d96928" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 46057 } } peers { permanent_uuid: "063967dd04ee4fec97fccb5c6a5815f8" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 33411 } }
I20260812 02:31:53.461580 17329 raft_consensus.cc:374] T a5586833c799404ca735aea68923b550 P 063967dd04ee4fec97fccb5c6a5815f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:31:53.461620 17329 raft_consensus.cc:729] T a5586833c799404ca735aea68923b550 P 063967dd04ee4fec97fccb5c6a5815f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 063967dd04ee4fec97fccb5c6a5815f8, State: Initialized, Role: FOLLOWER
I20260812 02:31:53.461583 17328 raft_consensus.cc:348] T a5586833c799404ca735aea68923b550 P f6f4b69dba6341ff926f2501f7d96928 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac78ac69480a4042a6278f187e3b4d61" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 38185 } } peers { permanent_uuid: "f6f4b69dba6341ff926f2501f7d96928" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 46057 } } peers { permanent_uuid: "063967dd04ee4fec97fccb5c6a5815f8" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 33411 } }
I20260812 02:31:53.461750 17328 raft_consensus.cc:374] T a5586833c799404ca735aea68923b550 P f6f4b69dba6341ff926f2501f7d96928 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:31:53.461794 17328 raft_consensus.cc:729] T a5586833c799404ca735aea68923b550 P f6f4b69dba6341ff926f2501f7d96928 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6f4b69dba6341ff926f2501f7d96928, State: Initialized, Role: FOLLOWER
I20260812 02:31:53.461814 17329 consensus_queue.cc:260] T a5586833c799404ca735aea68923b550 P 063967dd04ee4fec97fccb5c6a5815f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac78ac69480a4042a6278f187e3b4d61" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 38185 } } peers { permanent_uuid: "f6f4b69dba6341ff926f2501f7d96928" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 46057 } } peers { permanent_uuid: "063967dd04ee4fec97fccb5c6a5815f8" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 33411 } }
I20260812 02:31:53.462001 17328 consensus_queue.cc:260] T a5586833c799404ca735aea68923b550 P f6f4b69dba6341ff926f2501f7d96928 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac78ac69480a4042a6278f187e3b4d61" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 38185 } } peers { permanent_uuid: "f6f4b69dba6341ff926f2501f7d96928" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 46057 } } peers { permanent_uuid: "063967dd04ee4fec97fccb5c6a5815f8" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 33411 } }
I20260812 02:31:53.462200 17329 ts_tablet_manager.cc:1428] T a5586833c799404ca735aea68923b550 P 063967dd04ee4fec97fccb5c6a5815f8: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260812 02:31:53.462410 17328 ts_tablet_manager.cc:1428] T a5586833c799404ca735aea68923b550 P f6f4b69dba6341ff926f2501f7d96928: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260812 02:31:53.463995 17327 tablet_bootstrap.cc:492] T a5586833c799404ca735aea68923b550 P ac78ac69480a4042a6278f187e3b4d61: No bootstrap required, opened a new log
I20260812 02:31:53.464118 17327 ts_tablet_manager.cc:1397] T a5586833c799404ca735aea68923b550 P ac78ac69480a4042a6278f187e3b4d61: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260812 02:31:53.465159 17327 raft_consensus.cc:348] T a5586833c799404ca735aea68923b550 P ac78ac69480a4042a6278f187e3b4d61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac78ac69480a4042a6278f187e3b4d61" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 38185 } } peers { permanent_uuid: "f6f4b69dba6341ff926f2501f7d96928" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 46057 } } peers { permanent_uuid: "063967dd04ee4fec97fccb5c6a5815f8" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 33411 } }
I20260812 02:31:53.465342 17327 raft_consensus.cc:374] T a5586833c799404ca735aea68923b550 P ac78ac69480a4042a6278f187e3b4d61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:31:53.465389 17327 raft_consensus.cc:729] T a5586833c799404ca735aea68923b550 P ac78ac69480a4042a6278f187e3b4d61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac78ac69480a4042a6278f187e3b4d61, State: Initialized, Role: FOLLOWER
I20260812 02:31:53.465550 17327 consensus_queue.cc:260] T a5586833c799404ca735aea68923b550 P ac78ac69480a4042a6278f187e3b4d61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac78ac69480a4042a6278f187e3b4d61" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 38185 } } peers { permanent_uuid: "f6f4b69dba6341ff926f2501f7d96928" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 46057 } } peers { permanent_uuid: "063967dd04ee4fec97fccb5c6a5815f8" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 33411 } }
I20260812 02:31:53.466032 17327 ts_tablet_manager.cc:1428] T a5586833c799404ca735aea68923b550 P ac78ac69480a4042a6278f187e3b4d61: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260812 02:31:53.501019 17333 raft_consensus.cc:482] T a5586833c799404ca735aea68923b550 P ac78ac69480a4042a6278f187e3b4d61 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 02:31:53.501219 17333 raft_consensus.cc:504] T a5586833c799404ca735aea68923b550 P ac78ac69480a4042a6278f187e3b4d61 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac78ac69480a4042a6278f187e3b4d61" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 38185 } } peers { permanent_uuid: "f6f4b69dba6341ff926f2501f7d96928" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 46057 } } peers { permanent_uuid: "063967dd04ee4fec97fccb5c6a5815f8" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 33411 } }
I20260812 02:31:53.503440 17333 leader_election.cc:288] T a5586833c799404ca735aea68923b550 P ac78ac69480a4042a6278f187e3b4d61 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f6f4b69dba6341ff926f2501f7d96928 (127.14.103.2:46057), 063967dd04ee4fec97fccb5c6a5815f8 (127.14.103.3:33411)
I20260812 02:31:53.503650 17273 raft_consensus.cc:482] T a5586833c799404ca735aea68923b550 P f6f4b69dba6341ff926f2501f7d96928 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 02:31:53.503916 17273 raft_consensus.cc:504] T a5586833c799404ca735aea68923b550 P f6f4b69dba6341ff926f2501f7d96928 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac78ac69480a4042a6278f187e3b4d61" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 38185 } } peers { permanent_uuid: "f6f4b69dba6341ff926f2501f7d96928" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 46057 } } peers { permanent_uuid: "063967dd04ee4fec97fccb5c6a5815f8" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 33411 } }
I20260812 02:31:53.504531 17273 leader_election.cc:288] T a5586833c799404ca735aea68923b550 P f6f4b69dba6341ff926f2501f7d96928 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac78ac69480a4042a6278f187e3b4d61 (127.14.103.1:38185), 063967dd04ee4fec97fccb5c6a5815f8 (127.14.103.3:33411)
I20260812 02:31:53.504945 17142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5586833c799404ca735aea68923b550" candidate_uuid: "f6f4b69dba6341ff926f2501f7d96928" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "063967dd04ee4fec97fccb5c6a5815f8" is_pre_election: true
I20260812 02:31:53.505136 17142 raft_consensus.cc:2454] T a5586833c799404ca735aea68923b550 P 063967dd04ee4fec97fccb5c6a5815f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6f4b69dba6341ff926f2501f7d96928 in term 0.
I20260812 02:31:53.505107 16835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5586833c799404ca735aea68923b550" candidate_uuid: "f6f4b69dba6341ff926f2501f7d96928" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac78ac69480a4042a6278f187e3b4d61" is_pre_election: true
I20260812 02:31:53.505334 16835 raft_consensus.cc:2454] T a5586833c799404ca735aea68923b550 P ac78ac69480a4042a6278f187e3b4d61 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6f4b69dba6341ff926f2501f7d96928 in term 0.
I20260812 02:31:53.505801 16916 leader_election.cc:302] T a5586833c799404ca735aea68923b550 P f6f4b69dba6341ff926f2501f7d96928 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac78ac69480a4042a6278f187e3b4d61, f6f4b69dba6341ff926f2501f7d96928; no voters:
I20260812 02:31:53.506050 17273 raft_consensus.cc:2781] T a5586833c799404ca735aea68923b550 P f6f4b69dba6341ff926f2501f7d96928 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 02:31:53.506124 17273 raft_consensus.cc:482] T a5586833c799404ca735aea68923b550 P f6f4b69dba6341ff926f2501f7d96928 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 02:31:53.506166 17273 raft_consensus.cc:3037] T a5586833c799404ca735aea68923b550 P f6f4b69dba6341ff926f2501f7d96928 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:31:53.507710 17273 raft_consensus.cc:504] T a5586833c799404ca735aea68923b550 P f6f4b69dba6341ff926f2501f7d96928 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac78ac69480a4042a6278f187e3b4d61" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 38185 } } peers { permanent_uuid: "f6f4b69dba6341ff926f2501f7d96928" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 46057 } } peers { permanent_uuid: "063967dd04ee4fec97fccb5c6a5815f8" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 33411 } }
I20260812 02:31:53.508208 17273 leader_election.cc:288] T a5586833c799404ca735aea68923b550 P f6f4b69dba6341ff926f2501f7d96928 [CANDIDATE]: Term 1 election: Requested vote from peers ac78ac69480a4042a6278f187e3b4d61 (127.14.103.1:38185), 063967dd04ee4fec97fccb5c6a5815f8 (127.14.103.3:33411)
I20260812 02:31:53.508611 16835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5586833c799404ca735aea68923b550" candidate_uuid: "f6f4b69dba6341ff926f2501f7d96928" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac78ac69480a4042a6278f187e3b4d61"
I20260812 02:31:53.508653 17142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5586833c799404ca735aea68923b550" candidate_uuid: "f6f4b69dba6341ff926f2501f7d96928" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "063967dd04ee4fec97fccb5c6a5815f8"
I20260812 02:31:53.508750 16835 raft_consensus.cc:3037] T a5586833c799404ca735aea68923b550 P ac78ac69480a4042a6278f187e3b4d61 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:31:53.508772 17142 raft_consensus.cc:3037] T a5586833c799404ca735aea68923b550 P 063967dd04ee4fec97fccb5c6a5815f8 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:31:53.509130 17336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:31:53.503616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.2:46057 (local address 127.14.103.1:39543)
0812 02:31:53.504334 (+ 718us) negotiation.cc:107] Waiting for socket to connect
0812 02:31:53.504352 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 02:31:53.504468 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:31:53.504890 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:31:53.504894 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:31:53.504912 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:31:53.505062 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:31:53.505067 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:31:53.506158 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:31:53.506162 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:31:53.507572 (+ 1410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:31:53.507587 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:31:53.508653 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:31:53.508657 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:31:53.508756 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:31:53.508782 (+ 26us) client_negotiation.cc:769] Sending connection context
0812 02:31:53.508855 (+ 73us) client_negotiation.cc:240] Negotiation successful
0812 02:31:53.508951 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":616,"mutex_wait_us":33,"thread_start_us":195,"threads_started":1}
I20260812 02:31:53.509557 17335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:31:53.503897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.1:39543 (local address 127.14.103.2:46057)
0812 02:31:53.504451 (+ 554us) server_negotiation.cc:206] Beginning negotiation
0812 02:31:53.504466 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0812 02:31:53.504481 (+ 15us) server_negotiation.cc:407] Connection header received
0812 02:31:53.504621 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:31:53.504625 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:31:53.504681 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:31:53.504757 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:31:53.505223 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:31:53.506018 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:31:53.507731 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:31:53.508379 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:31:53.508524 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:31:53.508590 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0812 02:31:53.509305 (+ 715us) server_negotiation.cc:299] Negotiation successful
0812 02:31:53.509411 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":483,"thread_start_us":117,"threads_started":1}
I20260812 02:31:53.510013 16835 raft_consensus.cc:2454] T a5586833c799404ca735aea68923b550 P ac78ac69480a4042a6278f187e3b4d61 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6f4b69dba6341ff926f2501f7d96928 in term 1.
I20260812 02:31:53.510015 17142 raft_consensus.cc:2454] T a5586833c799404ca735aea68923b550 P 063967dd04ee4fec97fccb5c6a5815f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6f4b69dba6341ff926f2501f7d96928 in term 1.
I20260812 02:31:53.510066 17334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:31:53.503999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.3:33411 (local address 127.14.103.1:51889)
0812 02:31:53.504490 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0812 02:31:53.504510 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 02:31:53.504712 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:31:53.506061 (+ 1349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:31:53.506063 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:31:53.506083 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:31:53.506224 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:31:53.506230 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:31:53.507157 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:31:53.507161 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:31:53.508344 (+ 1183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:31:53.508357 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:31:53.509459 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:31:53.509464 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:31:53.509575 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:31:53.509606 (+ 31us) client_negotiation.cc:769] Sending connection context
0812 02:31:53.509687 (+ 81us) client_negotiation.cc:240] Negotiation successful
0812 02:31:53.509938 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":393,"mutex_wait_us":1,"thread_start_us":95,"threads_started":1}
I20260812 02:31:53.510571 17337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:31:53.504149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.1:51889 (local address 127.14.103.3:33411)
0812 02:31:53.505693 (+ 1544us) server_negotiation.cc:206] Beginning negotiation
0812 02:31:53.505715 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0812 02:31:53.505732 (+ 17us) server_negotiation.cc:407] Connection header received
0812 02:31:53.505788 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:31:53.505791 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:31:53.505845 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:31:53.505928 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:31:53.506384 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:31:53.507017 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:31:53.508532 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:31:53.509201 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:31:53.509323 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:31:53.509373 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0812 02:31:53.510282 (+ 909us) server_negotiation.cc:299] Negotiation successful
0812 02:31:53.510430 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":768,"thread_start_us":113,"threads_started":1}
I20260812 02:31:53.510564 16914 leader_election.cc:302] T a5586833c799404ca735aea68923b550 P f6f4b69dba6341ff926f2501f7d96928 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 063967dd04ee4fec97fccb5c6a5815f8, f6f4b69dba6341ff926f2501f7d96928; no voters:
I20260812 02:31:53.510792 17273 raft_consensus.cc:2781] T a5586833c799404ca735aea68923b550 P f6f4b69dba6341ff926f2501f7d96928 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:31:53.510887 17273 raft_consensus.cc:686] T a5586833c799404ca735aea68923b550 P f6f4b69dba6341ff926f2501f7d96928 [term 1 LEADER]: Becoming Leader. State: Replica: f6f4b69dba6341ff926f2501f7d96928, State: Running, Role: LEADER
I20260812 02:31:53.511104 17142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5586833c799404ca735aea68923b550" candidate_uuid: "ac78ac69480a4042a6278f187e3b4d61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "063967dd04ee4fec97fccb5c6a5815f8" is_pre_election: true
I20260812 02:31:53.511133 16989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5586833c799404ca735aea68923b550" candidate_uuid: "ac78ac69480a4042a6278f187e3b4d61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6f4b69dba6341ff926f2501f7d96928" is_pre_election: true
I20260812 02:31:53.511289 17142 raft_consensus.cc:2379] T a5586833c799404ca735aea68923b550 P 063967dd04ee4fec97fccb5c6a5815f8 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ac78ac69480a4042a6278f187e3b4d61 in current term 1: Already voted for candidate f6f4b69dba6341ff926f2501f7d96928 in this term.
I20260812 02:31:53.511602 17273 consensus_queue.cc:237] T a5586833c799404ca735aea68923b550 P f6f4b69dba6341ff926f2501f7d96928 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac78ac69480a4042a6278f187e3b4d61" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 38185 } } peers { permanent_uuid: "f6f4b69dba6341ff926f2501f7d96928" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 46057 } } peers { permanent_uuid: "063967dd04ee4fec97fccb5c6a5815f8" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 33411 } }
I20260812 02:31:53.511916 16762 leader_election.cc:302] T a5586833c799404ca735aea68923b550 P ac78ac69480a4042a6278f187e3b4d61 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac78ac69480a4042a6278f187e3b4d61; no voters: 063967dd04ee4fec97fccb5c6a5815f8, f6f4b69dba6341ff926f2501f7d96928
I20260812 02:31:53.512354 17333 raft_consensus.cc:2726] T a5586833c799404ca735aea68923b550 P ac78ac69480a4042a6278f187e3b4d61 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260812 02:31:53.515122 16664 catalog_manager.cc:5495] T a5586833c799404ca735aea68923b550 P f6f4b69dba6341ff926f2501f7d96928 reported cstate change: term changed from 0 to 1, leader changed from <none> to f6f4b69dba6341ff926f2501f7d96928 (127.14.103.2). New cstate: current_term: 1 leader_uuid: "f6f4b69dba6341ff926f2501f7d96928" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac78ac69480a4042a6278f187e3b4d61" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 38185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f6f4b69dba6341ff926f2501f7d96928" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 46057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "063967dd04ee4fec97fccb5c6a5815f8" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 33411 } health_report { overall_health: UNKNOWN } } }
I20260812 02:31:53.542160 17335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:31:53.537815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55200 (local address 127.14.103.2:46057)
0812 02:31:53.537934 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0812 02:31:53.537938 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:31:53.537999 (+ 61us) server_negotiation.cc:407] Connection header received
0812 02:31:53.538330 (+ 331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:31:53.538333 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:31:53.538387 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:31:53.538461 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 02:31:53.538892 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:31:53.539513 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:31:53.540684 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:31:53.540957 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:31:53.541054 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:31:53.541611 (+ 557us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 02:31:53.541717 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 02:31:53.541787 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0812 02:31:53.541903 (+ 116us) server_negotiation.cc:299] Negotiation successful
0812 02:31:53.541974 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260812 02:31:53.550694 17142 raft_consensus.cc:1261] T a5586833c799404ca735aea68923b550 P 063967dd04ee4fec97fccb5c6a5815f8 [term 1 FOLLOWER]: Refusing update from remote peer f6f4b69dba6341ff926f2501f7d96928: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:31:53.550788 16835 raft_consensus.cc:1261] T a5586833c799404ca735aea68923b550 P ac78ac69480a4042a6278f187e3b4d61 [term 1 FOLLOWER]: Refusing update from remote peer f6f4b69dba6341ff926f2501f7d96928: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:31:53.551235 17273 consensus_queue.cc:1035] T a5586833c799404ca735aea68923b550 P f6f4b69dba6341ff926f2501f7d96928 [LEADER]: Connected to new peer: Peer: permanent_uuid: "063967dd04ee4fec97fccb5c6a5815f8" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 33411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:31:53.551515 17308 consensus_queue.cc:1035] T a5586833c799404ca735aea68923b550 P f6f4b69dba6341ff926f2501f7d96928 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac78ac69480a4042a6278f187e3b4d61" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 38185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:31:53.590435 16659 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51194:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 02:31:53.590611 16659 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51194:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 02:31:53.593591 16659 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 43b9d054e04b419f9b79d092ff15ba6b: Sending DeleteTablet for 3 replicas of tablet a5586833c799404ca735aea68923b550
I20260812 02:31:53.594775 16814 tablet_service.cc:1514] Processing DeleteTablet for tablet a5586833c799404ca735aea68923b550 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:31:53 UTC) from {username='kudu', principal='kudu/127.14.103.62@KRBTEST.COM'} at 127.0.0.1:38290
I20260812 02:31:53.594852 16966 tablet_service.cc:1514] Processing DeleteTablet for tablet a5586833c799404ca735aea68923b550 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:31:53 UTC) from {username='kudu', principal='kudu/127.14.103.62@KRBTEST.COM'} at 127.0.0.1:55176
I20260812 02:31:53.595119 17316 tablet_replica.cc:330] T a5586833c799404ca735aea68923b550 P f6f4b69dba6341ff926f2501f7d96928: stopping tablet replica
I20260812 02:31:53.595142 17315 tablet_replica.cc:330] T a5586833c799404ca735aea68923b550 P ac78ac69480a4042a6278f187e3b4d61: stopping tablet replica
I20260812 02:31:53.595108 17118 tablet_service.cc:1514] Processing DeleteTablet for tablet a5586833c799404ca735aea68923b550 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:31:53 UTC) from {username='kudu', principal='kudu/127.14.103.62@KRBTEST.COM'} at 127.0.0.1:57520
I20260812 02:31:53.595271 17316 raft_consensus.cc:2229] T a5586833c799404ca735aea68923b550 P f6f4b69dba6341ff926f2501f7d96928 [term 1 LEADER]: Raft consensus shutting down.
I20260812 02:31:53.595278 17315 raft_consensus.cc:2229] T a5586833c799404ca735aea68923b550 P ac78ac69480a4042a6278f187e3b4d61 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 02:31:53.595312 17314 tablet_replica.cc:330] T a5586833c799404ca735aea68923b550 P 063967dd04ee4fec97fccb5c6a5815f8: stopping tablet replica
I20260812 02:31:53.595515 17315 raft_consensus.cc:2258] T a5586833c799404ca735aea68923b550 P ac78ac69480a4042a6278f187e3b4d61 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:31:53.595506 17314 raft_consensus.cc:2229] T a5586833c799404ca735aea68923b550 P 063967dd04ee4fec97fccb5c6a5815f8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 02:31:53.595566 17316 raft_consensus.cc:2258] T a5586833c799404ca735aea68923b550 P f6f4b69dba6341ff926f2501f7d96928 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:31:53.595650 17314 raft_consensus.cc:2258] T a5586833c799404ca735aea68923b550 P 063967dd04ee4fec97fccb5c6a5815f8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:31:53.596269 16659 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51194:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260812 02:31:53.596997 17315 ts_tablet_manager.cc:1905] T a5586833c799404ca735aea68923b550 P ac78ac69480a4042a6278f187e3b4d61: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:31:53.597007 17316 ts_tablet_manager.cc:1905] T a5586833c799404ca735aea68923b550 P f6f4b69dba6341ff926f2501f7d96928: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:31:53.597363 17314 ts_tablet_manager.cc:1905] T a5586833c799404ca735aea68923b550 P 063967dd04ee4fec97fccb5c6a5815f8: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260812 02:31:53.597424 16659 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 02:31:53.601313 17314 ts_tablet_manager.cc:1918] T a5586833c799404ca735aea68923b550 P 063967dd04ee4fec97fccb5c6a5815f8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 02:31:53.601451 17314 log.cc:1196] T a5586833c799404ca735aea68923b550 P 063967dd04ee4fec97fccb5c6a5815f8: Deleting WAL directory at /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501898713699-14748-0/minicluster-data/ts-2/wal/wals/a5586833c799404ca735aea68923b550
I20260812 02:31:53.601696 17316 ts_tablet_manager.cc:1918] T a5586833c799404ca735aea68923b550 P f6f4b69dba6341ff926f2501f7d96928: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 02:31:53.601792 17316 log.cc:1196] T a5586833c799404ca735aea68923b550 P f6f4b69dba6341ff926f2501f7d96928: Deleting WAL directory at /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501898713699-14748-0/minicluster-data/ts-1/wal/wals/a5586833c799404ca735aea68923b550
I20260812 02:31:53.602103 17316 ts_tablet_manager.cc:1939] T a5586833c799404ca735aea68923b550 P f6f4b69dba6341ff926f2501f7d96928: Deleting consensus metadata
I20260812 02:31:53.603981 16649 catalog_manager.cc:4841] TS f6f4b69dba6341ff926f2501f7d96928 (127.14.103.2:46057): tablet a5586833c799404ca735aea68923b550 (table test-table [id=f9f0c51b99334a02803efecf69e837dd]) successfully deleted
I20260812 02:31:53.602949 17314 ts_tablet_manager.cc:1939] T a5586833c799404ca735aea68923b550 P 063967dd04ee4fec97fccb5c6a5815f8: Deleting consensus metadata
I20260812 02:31:53.605105 16647 catalog_manager.cc:4841] TS 063967dd04ee4fec97fccb5c6a5815f8 (127.14.103.3:33411): tablet a5586833c799404ca735aea68923b550 (table test-table [id=f9f0c51b99334a02803efecf69e837dd]) successfully deleted
I20260812 02:31:53.606864 17315 ts_tablet_manager.cc:1918] T a5586833c799404ca735aea68923b550 P ac78ac69480a4042a6278f187e3b4d61: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 02:31:53.606999 17315 log.cc:1196] T a5586833c799404ca735aea68923b550 P ac78ac69480a4042a6278f187e3b4d61: Deleting WAL directory at /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501898713699-14748-0/minicluster-data/ts-0/wal/wals/a5586833c799404ca735aea68923b550
I20260812 02:31:53.607481 17315 ts_tablet_manager.cc:1939] T a5586833c799404ca735aea68923b550 P ac78ac69480a4042a6278f187e3b4d61: Deleting consensus metadata
I20260812 02:31:53.608381 16966 tablet_service.cc:1467] Processing CreateTablet for tablet 204ccda7acc743a28e3772ac8f218616 (DEFAULT_TABLE table=test-table [id=3f4d176d638546d38b5771ca7d6a922d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:31:53.608381 16814 tablet_service.cc:1467] Processing CreateTablet for tablet 204ccda7acc743a28e3772ac8f218616 (DEFAULT_TABLE table=test-table [id=3f4d176d638546d38b5771ca7d6a922d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:31:53.608577 17118 tablet_service.cc:1467] Processing CreateTablet for tablet 204ccda7acc743a28e3772ac8f218616 (DEFAULT_TABLE table=test-table [id=3f4d176d638546d38b5771ca7d6a922d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:31:53.608705 16814 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 204ccda7acc743a28e3772ac8f218616. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:31:53.608705 16966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 204ccda7acc743a28e3772ac8f218616. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:31:53.608876 17118 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 204ccda7acc743a28e3772ac8f218616. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:31:53.609408 16649 catalog_manager.cc:4841] TS ac78ac69480a4042a6278f187e3b4d61 (127.14.103.1:38185): tablet a5586833c799404ca735aea68923b550 (table test-table [id=f9f0c51b99334a02803efecf69e837dd]) successfully deleted
I20260812 02:31:53.611999 17329 tablet_bootstrap.cc:492] T 204ccda7acc743a28e3772ac8f218616 P 063967dd04ee4fec97fccb5c6a5815f8: Bootstrap starting.
I20260812 02:31:53.612141 17328 tablet_bootstrap.cc:492] T 204ccda7acc743a28e3772ac8f218616 P f6f4b69dba6341ff926f2501f7d96928: Bootstrap starting.
I20260812 02:31:53.612931 17328 tablet_bootstrap.cc:654] T 204ccda7acc743a28e3772ac8f218616 P f6f4b69dba6341ff926f2501f7d96928: Neither blocks nor log segments found. Creating new log.
I20260812 02:31:53.613008 17329 tablet_bootstrap.cc:654] T 204ccda7acc743a28e3772ac8f218616 P 063967dd04ee4fec97fccb5c6a5815f8: Neither blocks nor log segments found. Creating new log.
I20260812 02:31:53.613611 17327 tablet_bootstrap.cc:492] T 204ccda7acc743a28e3772ac8f218616 P ac78ac69480a4042a6278f187e3b4d61: Bootstrap starting.
I20260812 02:31:53.614188 17328 tablet_bootstrap.cc:492] T 204ccda7acc743a28e3772ac8f218616 P f6f4b69dba6341ff926f2501f7d96928: No bootstrap required, opened a new log
I20260812 02:31:53.614261 17328 ts_tablet_manager.cc:1397] T 204ccda7acc743a28e3772ac8f218616 P f6f4b69dba6341ff926f2501f7d96928: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260812 02:31:53.614508 17327 tablet_bootstrap.cc:654] T 204ccda7acc743a28e3772ac8f218616 P ac78ac69480a4042a6278f187e3b4d61: Neither blocks nor log segments found. Creating new log.
I20260812 02:31:53.614599 17328 raft_consensus.cc:348] T 204ccda7acc743a28e3772ac8f218616 P f6f4b69dba6341ff926f2501f7d96928 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac78ac69480a4042a6278f187e3b4d61" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 38185 } } peers { permanent_uuid: "f6f4b69dba6341ff926f2501f7d96928" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 46057 } } peers { permanent_uuid: "063967dd04ee4fec97fccb5c6a5815f8" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 33411 } }
I20260812 02:31:53.614693 17328 raft_consensus.cc:374] T 204ccda7acc743a28e3772ac8f218616 P f6f4b69dba6341ff926f2501f7d96928 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:31:53.614715 17328 raft_consensus.cc:729] T 204ccda7acc743a28e3772ac8f218616 P f6f4b69dba6341ff926f2501f7d96928 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6f4b69dba6341ff926f2501f7d96928, State: Initialized, Role: FOLLOWER
I20260812 02:31:53.614861 17328 consensus_queue.cc:260] T 204ccda7acc743a28e3772ac8f218616 P f6f4b69dba6341ff926f2501f7d96928 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac78ac69480a4042a6278f187e3b4d61" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 38185 } } peers { permanent_uuid: "f6f4b69dba6341ff926f2501f7d96928" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 46057 } } peers { permanent_uuid: "063967dd04ee4fec97fccb5c6a5815f8" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 33411 } }
I20260812 02:31:53.615329 17328 ts_tablet_manager.cc:1428] T 204ccda7acc743a28e3772ac8f218616 P f6f4b69dba6341ff926f2501f7d96928: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260812 02:31:53.615924 17327 tablet_bootstrap.cc:492] T 204ccda7acc743a28e3772ac8f218616 P ac78ac69480a4042a6278f187e3b4d61: No bootstrap required, opened a new log
I20260812 02:31:53.615926 17329 tablet_bootstrap.cc:492] T 204ccda7acc743a28e3772ac8f218616 P 063967dd04ee4fec97fccb5c6a5815f8: No bootstrap required, opened a new log
I20260812 02:31:53.616040 17329 ts_tablet_manager.cc:1397] T 204ccda7acc743a28e3772ac8f218616 P 063967dd04ee4fec97fccb5c6a5815f8: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260812 02:31:53.616040 17327 ts_tablet_manager.cc:1397] T 204ccda7acc743a28e3772ac8f218616 P ac78ac69480a4042a6278f187e3b4d61: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260812 02:31:53.616531 17327 raft_consensus.cc:348] T 204ccda7acc743a28e3772ac8f218616 P ac78ac69480a4042a6278f187e3b4d61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac78ac69480a4042a6278f187e3b4d61" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 38185 } } peers { permanent_uuid: "f6f4b69dba6341ff926f2501f7d96928" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 46057 } } peers { permanent_uuid: "063967dd04ee4fec97fccb5c6a5815f8" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 33411 } }
I20260812 02:31:53.616526 17329 raft_consensus.cc:348] T 204ccda7acc743a28e3772ac8f218616 P 063967dd04ee4fec97fccb5c6a5815f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac78ac69480a4042a6278f187e3b4d61" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 38185 } } peers { permanent_uuid: "f6f4b69dba6341ff926f2501f7d96928" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 46057 } } peers { permanent_uuid: "063967dd04ee4fec97fccb5c6a5815f8" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 33411 } }
I20260812 02:31:53.616930 17329 raft_consensus.cc:374] T 204ccda7acc743a28e3772ac8f218616 P 063967dd04ee4fec97fccb5c6a5815f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:31:53.617017 17329 raft_consensus.cc:729] T 204ccda7acc743a28e3772ac8f218616 P 063967dd04ee4fec97fccb5c6a5815f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 063967dd04ee4fec97fccb5c6a5815f8, State: Initialized, Role: FOLLOWER
I20260812 02:31:53.617002 17327 raft_consensus.cc:374] T 204ccda7acc743a28e3772ac8f218616 P ac78ac69480a4042a6278f187e3b4d61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:31:53.617063 17327 raft_consensus.cc:729] T 204ccda7acc743a28e3772ac8f218616 P ac78ac69480a4042a6278f187e3b4d61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac78ac69480a4042a6278f187e3b4d61, State: Initialized, Role: FOLLOWER
I20260812 02:31:53.617304 17329 consensus_queue.cc:260] T 204ccda7acc743a28e3772ac8f218616 P 063967dd04ee4fec97fccb5c6a5815f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac78ac69480a4042a6278f187e3b4d61" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 38185 } } peers { permanent_uuid: "f6f4b69dba6341ff926f2501f7d96928" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 46057 } } peers { permanent_uuid: "063967dd04ee4fec97fccb5c6a5815f8" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 33411 } }
I20260812 02:31:53.617327 17327 consensus_queue.cc:260] T 204ccda7acc743a28e3772ac8f218616 P ac78ac69480a4042a6278f187e3b4d61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac78ac69480a4042a6278f187e3b4d61" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 38185 } } peers { permanent_uuid: "f6f4b69dba6341ff926f2501f7d96928" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 46057 } } peers { permanent_uuid: "063967dd04ee4fec97fccb5c6a5815f8" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 33411 } }
I20260812 02:31:53.617630 17327 ts_tablet_manager.cc:1428] T 204ccda7acc743a28e3772ac8f218616 P ac78ac69480a4042a6278f187e3b4d61: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260812 02:31:53.617630 17329 ts_tablet_manager.cc:1428] T 204ccda7acc743a28e3772ac8f218616 P 063967dd04ee4fec97fccb5c6a5815f8: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260812 02:31:53.925097 17308 raft_consensus.cc:482] T 204ccda7acc743a28e3772ac8f218616 P f6f4b69dba6341ff926f2501f7d96928 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 02:31:53.925264 17308 raft_consensus.cc:504] T 204ccda7acc743a28e3772ac8f218616 P f6f4b69dba6341ff926f2501f7d96928 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac78ac69480a4042a6278f187e3b4d61" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 38185 } } peers { permanent_uuid: "f6f4b69dba6341ff926f2501f7d96928" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 46057 } } peers { permanent_uuid: "063967dd04ee4fec97fccb5c6a5815f8" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 33411 } }
I20260812 02:31:53.925839 17308 leader_election.cc:288] T 204ccda7acc743a28e3772ac8f218616 P f6f4b69dba6341ff926f2501f7d96928 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac78ac69480a4042a6278f187e3b4d61 (127.14.103.1:38185), 063967dd04ee4fec97fccb5c6a5815f8 (127.14.103.3:33411)
I20260812 02:31:53.926539 16835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "204ccda7acc743a28e3772ac8f218616" candidate_uuid: "f6f4b69dba6341ff926f2501f7d96928" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac78ac69480a4042a6278f187e3b4d61" is_pre_election: true
I20260812 02:31:53.926542 17142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "204ccda7acc743a28e3772ac8f218616" candidate_uuid: "f6f4b69dba6341ff926f2501f7d96928" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "063967dd04ee4fec97fccb5c6a5815f8" is_pre_election: true
I20260812 02:31:53.926720 17142 raft_consensus.cc:2454] T 204ccda7acc743a28e3772ac8f218616 P 063967dd04ee4fec97fccb5c6a5815f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6f4b69dba6341ff926f2501f7d96928 in term 0.
I20260812 02:31:53.926736 16835 raft_consensus.cc:2454] T 204ccda7acc743a28e3772ac8f218616 P ac78ac69480a4042a6278f187e3b4d61 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6f4b69dba6341ff926f2501f7d96928 in term 0.
I20260812 02:31:53.927074 16916 leader_election.cc:302] T 204ccda7acc743a28e3772ac8f218616 P f6f4b69dba6341ff926f2501f7d96928 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac78ac69480a4042a6278f187e3b4d61, f6f4b69dba6341ff926f2501f7d96928; no voters:
I20260812 02:31:53.927297 17308 raft_consensus.cc:2781] T 204ccda7acc743a28e3772ac8f218616 P f6f4b69dba6341ff926f2501f7d96928 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 02:31:53.927362 17308 raft_consensus.cc:482] T 204ccda7acc743a28e3772ac8f218616 P f6f4b69dba6341ff926f2501f7d96928 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 02:31:53.927409 17308 raft_consensus.cc:3037] T 204ccda7acc743a28e3772ac8f218616 P f6f4b69dba6341ff926f2501f7d96928 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:31:53.928520 17308 raft_consensus.cc:504] T 204ccda7acc743a28e3772ac8f218616 P f6f4b69dba6341ff926f2501f7d96928 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac78ac69480a4042a6278f187e3b4d61" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 38185 } } peers { permanent_uuid: "f6f4b69dba6341ff926f2501f7d96928" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 46057 } } peers { permanent_uuid: "063967dd04ee4fec97fccb5c6a5815f8" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 33411 } }
I20260812 02:31:53.928931 17308 leader_election.cc:288] T 204ccda7acc743a28e3772ac8f218616 P f6f4b69dba6341ff926f2501f7d96928 [CANDIDATE]: Term 1 election: Requested vote from peers ac78ac69480a4042a6278f187e3b4d61 (127.14.103.1:38185), 063967dd04ee4fec97fccb5c6a5815f8 (127.14.103.3:33411)
I20260812 02:31:53.929718 16835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "204ccda7acc743a28e3772ac8f218616" candidate_uuid: "f6f4b69dba6341ff926f2501f7d96928" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac78ac69480a4042a6278f187e3b4d61"
I20260812 02:31:53.929868 16835 raft_consensus.cc:3037] T 204ccda7acc743a28e3772ac8f218616 P ac78ac69480a4042a6278f187e3b4d61 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:31:53.929857 17142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "204ccda7acc743a28e3772ac8f218616" candidate_uuid: "f6f4b69dba6341ff926f2501f7d96928" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "063967dd04ee4fec97fccb5c6a5815f8"
I20260812 02:31:53.930198 17142 raft_consensus.cc:3037] T 204ccda7acc743a28e3772ac8f218616 P 063967dd04ee4fec97fccb5c6a5815f8 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:31:53.931226 16835 raft_consensus.cc:2454] T 204ccda7acc743a28e3772ac8f218616 P ac78ac69480a4042a6278f187e3b4d61 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6f4b69dba6341ff926f2501f7d96928 in term 1.
I20260812 02:31:53.931468 17142 raft_consensus.cc:2454] T 204ccda7acc743a28e3772ac8f218616 P 063967dd04ee4fec97fccb5c6a5815f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6f4b69dba6341ff926f2501f7d96928 in term 1.
I20260812 02:31:53.931617 16916 leader_election.cc:302] T 204ccda7acc743a28e3772ac8f218616 P f6f4b69dba6341ff926f2501f7d96928 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac78ac69480a4042a6278f187e3b4d61, f6f4b69dba6341ff926f2501f7d96928; no voters:
I20260812 02:31:53.931854 17308 raft_consensus.cc:2781] T 204ccda7acc743a28e3772ac8f218616 P f6f4b69dba6341ff926f2501f7d96928 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:31:53.931977 17308 raft_consensus.cc:686] T 204ccda7acc743a28e3772ac8f218616 P f6f4b69dba6341ff926f2501f7d96928 [term 1 LEADER]: Becoming Leader. State: Replica: f6f4b69dba6341ff926f2501f7d96928, State: Running, Role: LEADER
I20260812 02:31:53.932126 17308 consensus_queue.cc:237] T 204ccda7acc743a28e3772ac8f218616 P f6f4b69dba6341ff926f2501f7d96928 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac78ac69480a4042a6278f187e3b4d61" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 38185 } } peers { permanent_uuid: "f6f4b69dba6341ff926f2501f7d96928" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 46057 } } peers { permanent_uuid: "063967dd04ee4fec97fccb5c6a5815f8" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 33411 } }
I20260812 02:31:53.934523 16659 catalog_manager.cc:5495] T 204ccda7acc743a28e3772ac8f218616 P f6f4b69dba6341ff926f2501f7d96928 reported cstate change: term changed from 0 to 1, leader changed from <none> to f6f4b69dba6341ff926f2501f7d96928 (127.14.103.2). New cstate: current_term: 1 leader_uuid: "f6f4b69dba6341ff926f2501f7d96928" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac78ac69480a4042a6278f187e3b4d61" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 38185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f6f4b69dba6341ff926f2501f7d96928" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 46057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "063967dd04ee4fec97fccb5c6a5815f8" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 33411 } health_report { overall_health: UNKNOWN } } }
I20260812 02:31:54.014168 16835 raft_consensus.cc:1261] T 204ccda7acc743a28e3772ac8f218616 P ac78ac69480a4042a6278f187e3b4d61 [term 1 FOLLOWER]: Refusing update from remote peer f6f4b69dba6341ff926f2501f7d96928: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:31:54.014245 17142 raft_consensus.cc:1261] T 204ccda7acc743a28e3772ac8f218616 P 063967dd04ee4fec97fccb5c6a5815f8 [term 1 FOLLOWER]: Refusing update from remote peer f6f4b69dba6341ff926f2501f7d96928: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:31:54.014643 17308 consensus_queue.cc:1035] T 204ccda7acc743a28e3772ac8f218616 P f6f4b69dba6341ff926f2501f7d96928 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac78ac69480a4042a6278f187e3b4d61" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 38185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:31:54.014923 17305 consensus_queue.cc:1035] T 204ccda7acc743a28e3772ac8f218616 P f6f4b69dba6341ff926f2501f7d96928 [LEADER]: Connected to new peer: Peer: permanent_uuid: "063967dd04ee4fec97fccb5c6a5815f8" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 33411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260812 02:31:54.034775 17297 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260812 02:31:55.628393 16659 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51194:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 02:31:55.629002 16659 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51194:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 02:31:55.633311 16659 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 43b9d054e04b419f9b79d092ff15ba6b: Sending DeleteTablet for 3 replicas of tablet 204ccda7acc743a28e3772ac8f218616
I20260812 02:31:55.634402 16814 tablet_service.cc:1514] Processing DeleteTablet for tablet 204ccda7acc743a28e3772ac8f218616 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:31:55 UTC) from {username='kudu', principal='kudu/127.14.103.62@KRBTEST.COM'} at 127.0.0.1:38290
I20260812 02:31:55.634451 16966 tablet_service.cc:1514] Processing DeleteTablet for tablet 204ccda7acc743a28e3772ac8f218616 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:31:55 UTC) from {username='kudu', principal='kudu/127.14.103.62@KRBTEST.COM'} at 127.0.0.1:55176
I20260812 02:31:55.634443 17118 tablet_service.cc:1514] Processing DeleteTablet for tablet 204ccda7acc743a28e3772ac8f218616 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:31:55 UTC) from {username='kudu', principal='kudu/127.14.103.62@KRBTEST.COM'} at 127.0.0.1:57520
I20260812 02:31:55.634886 17374 tablet_replica.cc:330] T 204ccda7acc743a28e3772ac8f218616 P f6f4b69dba6341ff926f2501f7d96928: stopping tablet replica
I20260812 02:31:55.634874 17376 tablet_replica.cc:330] T 204ccda7acc743a28e3772ac8f218616 P 063967dd04ee4fec97fccb5c6a5815f8: stopping tablet replica
I20260812 02:31:55.635054 17374 raft_consensus.cc:2229] T 204ccda7acc743a28e3772ac8f218616 P f6f4b69dba6341ff926f2501f7d96928 [term 1 LEADER]: Raft consensus shutting down.
I20260812 02:31:55.635131 14748 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbep4M4/build/debug/bin/kudu with pid 16742
I20260812 02:31:55.641501 17374 raft_consensus.cc:2258] T 204ccda7acc743a28e3772ac8f218616 P f6f4b69dba6341ff926f2501f7d96928 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:31:55.644488 17374 ts_tablet_manager.cc:1905] T 204ccda7acc743a28e3772ac8f218616 P f6f4b69dba6341ff926f2501f7d96928: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:31:55.635200 17376 raft_consensus.cc:2229] T 204ccda7acc743a28e3772ac8f218616 P 063967dd04ee4fec97fccb5c6a5815f8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 02:31:55.644940 17376 raft_consensus.cc:2258] T 204ccda7acc743a28e3772ac8f218616 P 063967dd04ee4fec97fccb5c6a5815f8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:31:55.646054 17376 ts_tablet_manager.cc:1905] T 204ccda7acc743a28e3772ac8f218616 P 063967dd04ee4fec97fccb5c6a5815f8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:31:55.649245 17374 ts_tablet_manager.cc:1918] T 204ccda7acc743a28e3772ac8f218616 P f6f4b69dba6341ff926f2501f7d96928: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260812 02:31:55.649489 17374 log.cc:1196] T 204ccda7acc743a28e3772ac8f218616 P f6f4b69dba6341ff926f2501f7d96928: Deleting WAL directory at /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501898713699-14748-0/minicluster-data/ts-1/wal/wals/204ccda7acc743a28e3772ac8f218616
I20260812 02:31:55.649922 17374 ts_tablet_manager.cc:1939] T 204ccda7acc743a28e3772ac8f218616 P f6f4b69dba6341ff926f2501f7d96928: Deleting consensus metadata
I20260812 02:31:55.650880 16649 catalog_manager.cc:4841] TS f6f4b69dba6341ff926f2501f7d96928 (127.14.103.2:46057): tablet 204ccda7acc743a28e3772ac8f218616 (table test-table [id=3f4d176d638546d38b5771ca7d6a922d]) successfully deleted
I20260812 02:31:55.651628 17376 ts_tablet_manager.cc:1918] T 204ccda7acc743a28e3772ac8f218616 P 063967dd04ee4fec97fccb5c6a5815f8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260812 02:31:55.651738 17376 log.cc:1196] T 204ccda7acc743a28e3772ac8f218616 P 063967dd04ee4fec97fccb5c6a5815f8: Deleting WAL directory at /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786501898713699-14748-0/minicluster-data/ts-2/wal/wals/204ccda7acc743a28e3772ac8f218616
I20260812 02:31:55.652033 17376 ts_tablet_manager.cc:1939] T 204ccda7acc743a28e3772ac8f218616 P 063967dd04ee4fec97fccb5c6a5815f8: Deleting consensus metadata
W20260812 02:31:55.653118 16649 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.103.1:38185 (error 108)
I20260812 02:31:55.653457 14748 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbep4M4/build/debug/bin/kudu with pid 16895
I20260812 02:31:55.658185 16647 catalog_manager.cc:4841] TS 063967dd04ee4fec97fccb5c6a5815f8 (127.14.103.3:33411): tablet 204ccda7acc743a28e3772ac8f218616 (table test-table [id=3f4d176d638546d38b5771ca7d6a922d]) successfully deleted
I20260812 02:31:55.660224 17378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:31:55.659433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.1:38185 (local address 127.0.0.1:38298)
0812 02:31:55.660017 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0812 02:31:55.660116 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.1:38185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":177,"threads_started":1}
W20260812 02:31:55.660421 16649 catalog_manager.cc:4584] TS ac78ac69480a4042a6278f187e3b4d61 (127.14.103.1:38185): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 204ccda7acc743a28e3772ac8f218616: Network error: Client connection negotiation failed: client connection to 127.14.103.1:38185: connect: Connection refused (error 111)
I20260812 02:31:55.667311 14748 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbep4M4/build/debug/bin/kudu with pid 17047
I20260812 02:31:55.680085 14748 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbep4M4/build/debug/bin/kudu with pid 16626
2026-08-12T02:31:55Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (7561 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786501898713699-14748-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 02:31:55 dist-test-slave-2w3w krb5kdc[17381](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 02:31:55 dist-test-slave-2w3w krb5kdc[17381](info): set up 2 sockets
Aug 12 02:31:55 dist-test-slave-2w3w krb5kdc[17381](info): commencing operation
krb5kdc: starting...
W20260812 02:31:57.063819 14748 mini_kdc.cc:121] Time spent starting KDC: real 1.352s user 0.012s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260812 02:31:57.331027 14748 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.267s user 0.004s sys 0.000s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260812 02:31:57.637372 14748 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.306s user 0.003s sys 0.000s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786501898713699-14748-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786501898713699-14748-0/krb5kdc/test-user.keytab.
Aug 12 02:31:57 dist-test-slave-2w3w krb5kdc[17381](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501917, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T02:31:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T02:31:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.103.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.103.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.103.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.62.keytab.
Entry for principal kudu/127.14.103.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.62.keytab.
I20260812 02:31:57.772428 14748 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbep4M4/build/debug/bin/kudu
/tmp/dist-test-taskbep4M4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786501898713699-14748-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786501898713699-14748-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786501898713699-14748-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786501898713699-14748-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.103.62:42887
--webserver_interface=127.14.103.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.62.keytab
--principal=kudu/127.14.103.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.103.20:46657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.103.62:42887
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786501898713699-14748-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786501898713699-14748-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786501898713699-14748-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:31:57.937070 17397 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:31:57.937898 17397 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:31:57.938045 17397 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:31:57.944804 17397 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:31:57.944965 17397 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:31:57.945013 17397 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:31:57.945060 17397 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:31:57.945235 17397 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 02:31:57.951817 17397 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.103.20:46657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786501898713699-14748-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786501898713699-14748-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.103.62:42887
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.62.keytab
--principal=kudu/127.14.103.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.103.62:42887
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786501898713699-14748-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.103.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17397
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786501898713699-14748-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:31:57.953121 17397 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:31:57.954324 17397 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260812 02:31:57.965116 17397 server_base.cc:1034] running on GCE node
W20260812 02:31:57.965189 17405 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:31:57.964998 17403 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:31:57.965512 17402 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:31:57.967046 17397 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:31:57.968483 17397 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:31:57.969705 17397 hybrid_clock.cc:648] HybridClock initialized: now 1786501917969659 us; error 66 us; skew 500 ppm
Aug 12 02:31:57 dist-test-slave-2w3w krb5kdc[17381](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501917, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:31:57.973788 17397 init.cc:377] Logged in from keytab as kudu/127.14.103.62@KRBTEST.COM (short username kudu)
I20260812 02:31:57.975147 17397 webserver.cc:460] Webserver started at http://127.14.103.62:34569/ using document root <none> and password file <none>
I20260812 02:31:57.975849 17397 fs_manager.cc:359] Metadata directory not provided
I20260812 02:31:57.975972 17397 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:31:57.976250 17397 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:31:57.980713 17397 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786501898713699-14748-0/minicluster-data/master-0/data/instance:
uuid: "7bb687531e1444f48940f2322942d4ff"
format_stamp: "Formatted at 2026-08-12 02:31:57 on dist-test-slave-2w3w"
server_key: "7425392173a4b8b30749fd572d96dbca"
server_key_iv: "90b30816fcc96d0012e1bbec2f963a3d"
server_key_version: "encryptionkey@0"
I20260812 02:31:57.981647 17397 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786501898713699-14748-0/minicluster-data/master-0/wal/instance:
uuid: "7bb687531e1444f48940f2322942d4ff"
format_stamp: "Formatted at 2026-08-12 02:31:57 on dist-test-slave-2w3w"
server_key: "7425392173a4b8b30749fd572d96dbca"
server_key_iv: "90b30816fcc96d0012e1bbec2f963a3d"
server_key_version: "encryptionkey@0"
I20260812 02:31:57.989068 17397 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.008s sys 0.000s
I20260812 02:31:57.992735 17414 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:31:57.994597 17397 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260812 02:31:57.994776 17397 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786501898713699-14748-0/minicluster-data/master-0/data,/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786501898713699-14748-0/minicluster-data/master-0/wal
uuid: "7bb687531e1444f48940f2322942d4ff"
format_stamp: "Formatted at 2026-08-12 02:31:57 on dist-test-slave-2w3w"
server_key: "7425392173a4b8b30749fd572d96dbca"
server_key_iv: "90b30816fcc96d0012e1bbec2f963a3d"
server_key_version: "encryptionkey@0"
I20260812 02:31:57.994900 17397 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786501898713699-14748-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786501898713699-14748-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786501898713699-14748-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:31:58.013530 17397 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:31:58.018915 17397 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:31:58.019515 17397 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:31:58.030889 17397 rpc_server.cc:307] RPC server started. Bound to: 127.14.103.62:42887
I20260812 02:31:58.031216 17470 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.103.62:42887 every 8 connection(s)
I20260812 02:31:58.032097 17397 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786501898713699-14748-0/minicluster-data/master-0/data/info.pb
I20260812 02:31:58.033632 14748 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbep4M4/build/debug/bin/kudu as pid 17397
I20260812 02:31:58.033782 14748 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786501898713699-14748-0/minicluster-data/master-0/wal/instance
I20260812 02:31:58.034154 14748 external_mini_cluster.cc:1428] Setting key 5e0f130b598e92992d63d77d07bcf1e0
I20260812 02:31:58.035948 17471 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:31:58.045441 17471 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7bb687531e1444f48940f2322942d4ff: Bootstrap starting.
Aug 12 02:31:58 dist-test-slave-2w3w krb5kdc[17381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501917, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.103.62@KRBTEST.COM
I20260812 02:31:58.048454 17471 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7bb687531e1444f48940f2322942d4ff: Neither blocks nor log segments found. Creating new log.
I20260812 02:31:58.049350 17471 log.cc:824] T 00000000000000000000000000000000 P 7bb687531e1444f48940f2322942d4ff: Log is configured to *not* fsync() on all Append() calls
I20260812 02:31:58.052994 17471 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7bb687531e1444f48940f2322942d4ff: No bootstrap required, opened a new log
I20260812 02:31:58.053879 17474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:31:58.036012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44988 (local address 127.14.103.62:42887)
0812 02:31:58.036651 (+ 639us) server_negotiation.cc:206] Beginning negotiation
0812 02:31:58.036668 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0812 02:31:58.036718 (+ 50us) server_negotiation.cc:407] Connection header received
0812 02:31:58.037418 (+ 700us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:31:58.037440 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:31:58.038348 (+ 908us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:31:58.038804 (+ 456us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:31:58.039844 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:31:58.040950 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:31:58.041906 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:31:58.042445 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:31:58.042643 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:31:58.046896 (+ 4253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:31:58.046905 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:31:58.046928 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:31:58.047061 (+ 133us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:31:58.049599 (+ 2538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:31:58.050246 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:31:58.050252 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:31:58.050259 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:31:58.050398 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:31:58.051019 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:31:58.051022 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:31:58.051024 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:31:58.051619 (+ 595us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:31:58.051872 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0812 02:31:58.052359 (+ 487us) server_negotiation.cc:299] Negotiation successful
0812 02:31:58.052795 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":458,"thread_start_us":176,"threads_started":1}
I20260812 02:31:58.058564 17471 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7bb687531e1444f48940f2322942d4ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bb687531e1444f48940f2322942d4ff" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 42887 } }
I20260812 02:31:58.058897 17471 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7bb687531e1444f48940f2322942d4ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:31:58.058948 17471 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7bb687531e1444f48940f2322942d4ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7bb687531e1444f48940f2322942d4ff, State: Initialized, Role: FOLLOWER
I20260812 02:31:58.059610 17471 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7bb687531e1444f48940f2322942d4ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bb687531e1444f48940f2322942d4ff" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 42887 } }
I20260812 02:31:58.059772 17471 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7bb687531e1444f48940f2322942d4ff [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 02:31:58.059823 17471 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7bb687531e1444f48940f2322942d4ff [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 02:31:58.059952 17471 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7bb687531e1444f48940f2322942d4ff [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:31:58.062065 17471 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7bb687531e1444f48940f2322942d4ff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bb687531e1444f48940f2322942d4ff" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 42887 } }
I20260812 02:31:58.062523 17471 leader_election.cc:302] T 00000000000000000000000000000000 P 7bb687531e1444f48940f2322942d4ff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7bb687531e1444f48940f2322942d4ff; no voters:
I20260812 02:31:58.062920 17471 leader_election.cc:288] T 00000000000000000000000000000000 P 7bb687531e1444f48940f2322942d4ff [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 02:31:58.063045 17476 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7bb687531e1444f48940f2322942d4ff [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:31:58.063270 17476 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7bb687531e1444f48940f2322942d4ff [term 1 LEADER]: Becoming Leader. State: Replica: 7bb687531e1444f48940f2322942d4ff, State: Running, Role: LEADER
I20260812 02:31:58.063638 17476 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7bb687531e1444f48940f2322942d4ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bb687531e1444f48940f2322942d4ff" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 42887 } }
I20260812 02:31:58.064505 17471 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7bb687531e1444f48940f2322942d4ff [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:31:58.066277 17478 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7bb687531e1444f48940f2322942d4ff [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7bb687531e1444f48940f2322942d4ff. Latest consensus state: current_term: 1 leader_uuid: "7bb687531e1444f48940f2322942d4ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bb687531e1444f48940f2322942d4ff" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 42887 } } }
I20260812 02:31:58.066521 17478 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7bb687531e1444f48940f2322942d4ff [sys.catalog]: This master's current role is: LEADER
I20260812 02:31:58.066843 17477 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7bb687531e1444f48940f2322942d4ff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7bb687531e1444f48940f2322942d4ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bb687531e1444f48940f2322942d4ff" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 42887 } } }
I20260812 02:31:58.067166 17477 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7bb687531e1444f48940f2322942d4ff [sys.catalog]: This master's current role is: LEADER
I20260812 02:31:58.067302 17486 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 02:31:58.070919 17486 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 02:31:58.078568 17486 catalog_manager.cc:1347] Generated new cluster ID: 47201c953a2b4a8ba0a24f833d4a3d77
I20260812 02:31:58.078704 17486 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 02:31:58.098277 17486 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 02:31:58.099960 17486 catalog_manager.cc:1504] Loading token signing keys...
I20260812 02:31:58.106084 17486 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7bb687531e1444f48940f2322942d4ff: Generated new TSK 0
I20260812 02:31:58.107179 17486 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.103.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.103.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.103.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.1.keytab.
Entry for principal kudu/127.14.103.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.1.keytab.
I20260812 02:31:58.153136 14748 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbep4M4/build/debug/bin/kudu
/tmp/dist-test-taskbep4M4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786501898713699-14748-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786501898713699-14748-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786501898713699-14748-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786501898713699-14748-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.103.1:0
--local_ip_for_outbound_sockets=127.14.103.1
--webserver_interface=127.14.103.1
--webserver_port=0
--tserver_master_addrs=127.14.103.62:42887
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.1.keytab
--principal=kudu/127.14.103.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.103.20:46657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786501898713699-14748-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786501898713699-14748-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786501898713699-14748-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:31:58.324533 17502 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:31:58.324995 17502 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:31:58.325080 17502 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:31:58.329908 17502 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:31:58.330042 17502 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:31:58.330230 17502 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.103.1
I20260812 02:31:58.337385 17502 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.103.20:46657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786501898713699-14748-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786501898713699-14748-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.1.keytab
--principal=kudu/127.14.103.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.103.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786501898713699-14748-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.103.1
--webserver_port=0
--tserver_master_addrs=127.14.103.62:42887
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17502
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.103.1
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786501898713699-14748-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:31:58.338704 17502 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:31:58.340276 17502 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:31:58.352847 17509 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:31:58.352785 17510 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:31:58.352917 17502 server_base.cc:1034] running on GCE node
W20260812 02:31:58.352802 17512 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:31:58.353663 17502 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:31:58.354393 17502 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:31:58.355599 17502 hybrid_clock.cc:648] HybridClock initialized: now 1786501918355547 us; error 61 us; skew 500 ppm
Aug 12 02:31:58 dist-test-slave-2w3w krb5kdc[17381](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501918, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:31:58.359542 17502 init.cc:377] Logged in from keytab as kudu/127.14.103.1@KRBTEST.COM (short username kudu)
I20260812 02:31:58.360643 17502 webserver.cc:460] Webserver started at http://127.14.103.1:33521/ using document root <none> and password file <none>
I20260812 02:31:58.361286 17502 fs_manager.cc:359] Metadata directory not provided
I20260812 02:31:58.361397 17502 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:31:58.361666 17502 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:31:58.364059 17502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786501898713699-14748-0/minicluster-data/ts-0/data/instance:
uuid: "4270a3bce37f4da7bf39ab045c3481c9"
format_stamp: "Formatted at 2026-08-12 02:31:58 on dist-test-slave-2w3w"
server_key: "e13dfb61a21061cf16a41875d752b92f"
server_key_iv: "a95bc5663d868f9ac60f1ca93fa0574f"
server_key_version: "encryptionkey@0"
I20260812 02:31:58.365294 17502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786501898713699-14748-0/minicluster-data/ts-0/wal/instance:
uuid: "4270a3bce37f4da7bf39ab045c3481c9"
format_stamp: "Formatted at 2026-08-12 02:31:58 on dist-test-slave-2w3w"
server_key: "e13dfb61a21061cf16a41875d752b92f"
server_key_iv: "a95bc5663d868f9ac60f1ca93fa0574f"
server_key_version: "encryptionkey@0"
I20260812 02:31:58.370159 17502 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260812 02:31:58.373056 17520 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:31:58.374504 17502 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260812 02:31:58.374696 17502 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786501898713699-14748-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786501898713699-14748-0/minicluster-data/ts-0/wal
uuid: "4270a3bce37f4da7bf39ab045c3481c9"
format_stamp: "Formatted at 2026-08-12 02:31:58 on dist-test-slave-2w3w"
server_key: "e13dfb61a21061cf16a41875d752b92f"
server_key_iv: "a95bc5663d868f9ac60f1ca93fa0574f"
server_key_version: "encryptionkey@0"
I20260812 02:31:58.374855 17502 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786501898713699-14748-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786501898713699-14748-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786501898713699-14748-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:31:58.395217 17502 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:31:58.399701 17502 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:31:58.399957 17502 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:31:58.400686 17502 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:31:58.401649 17502 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:31:58.401723 17502 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:31:58.401826 17502 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:31:58.401878 17502 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:31:58.411899 17502 rpc_server.cc:307] RPC server started. Bound to: 127.14.103.1:41299
I20260812 02:31:58.411923 17646 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.103.1:41299 every 8 connection(s)
I20260812 02:31:58.412946 17502 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786501898713699-14748-0/minicluster-data/ts-0/data/info.pb
I20260812 02:31:58.413889 14748 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbep4M4/build/debug/bin/kudu as pid 17502
I20260812 02:31:58.414036 14748 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786501898713699-14748-0/minicluster-data/ts-0/wal/instance
I20260812 02:31:58.414404 14748 external_mini_cluster.cc:1428] Setting key cb17d14b883a4be53c8e325ffd789305
WARNING: no policy specified for kudu/127.14.103.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.103.2@KRBTEST.COM" created.
Aug 12 02:31:58 dist-test-slave-2w3w krb5kdc[17381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501918, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.1@KRBTEST.COM for kudu/127.14.103.62@KRBTEST.COM
I20260812 02:31:58.447481 17474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:31:58.415202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.1:60259 (local address 127.14.103.62:42887)
0812 02:31:58.415345 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0812 02:31:58.415349 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:31:58.419071 (+ 3722us) server_negotiation.cc:407] Connection header received
0812 02:31:58.420583 (+ 1512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:31:58.420587 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:31:58.420643 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:31:58.420776 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:31:58.423775 (+ 2999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:31:58.424605 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:31:58.427266 (+ 2661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:31:58.427531 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:31:58.427698 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:31:58.435934 (+ 8236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:31:58.435938 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:31:58.435941 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:31:58.435989 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:31:58.443314 (+ 7325us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:31:58.444251 (+ 937us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:31:58.444256 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:31:58.444257 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:31:58.444370 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:31:58.444911 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:31:58.444915 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:31:58.444916 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:31:58.445544 (+ 628us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:31:58.445647 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0812 02:31:58.447162 (+ 1515us) server_negotiation.cc:299] Negotiation successful
0812 02:31:58.447334 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260812 02:31:58.448632 17650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:31:58.415646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:42887 (local address 127.14.103.1:60259)
0812 02:31:58.418859 (+ 3213us) negotiation.cc:107] Waiting for socket to connect
0812 02:31:58.418923 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0812 02:31:58.420325 (+ 1402us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:31:58.421065 (+ 740us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:31:58.421078 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:31:58.422264 (+ 1186us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:31:58.423450 (+ 1186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:31:58.423486 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:31:58.424813 (+ 1327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:31:58.424820 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:31:58.427017 (+ 2197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:31:58.427039 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:31:58.427779 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:31:58.427786 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:31:58.428058 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:31:58.428890 (+ 832us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:31:58.428913 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:31:58.435592 (+ 6679us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:31:58.443531 (+ 7939us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:31:58.443543 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:31:58.443568 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:31:58.444077 (+ 509us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:31:58.444486 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:31:58.444490 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:31:58.444492 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:31:58.444732 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:31:58.445811 (+ 1079us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:31:58.445822 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:31:58.446530 (+ 708us) client_negotiation.cc:769] Sending connection context
0812 02:31:58.446955 (+ 425us) client_negotiation.cc:240] Negotiation successful
0812 02:31:58.447433 (+ 478us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2910,"thread_start_us":161,"threads_started":1}
I20260812 02:31:58.450316 17647 heartbeater.cc:347] Connected to a master server at 127.14.103.62:42887
I20260812 02:31:58.450644 17647 heartbeater.cc:464] Registering TS with master...
I20260812 02:31:58.451316 17647 heartbeater.cc:511] Master 127.14.103.62:42887 requested a full tablet report, sending...
I20260812 02:31:58.453223 17432 ts_manager.cc:194] Registered new tserver with Master: 4270a3bce37f4da7bf39ab045c3481c9 (127.14.103.1:41299)
I20260812 02:31:58.454634 17432 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.103.1@KRBTEST.COM'} at 127.14.103.1:60259
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.103.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.2.keytab.
Entry for principal kudu/127.14.103.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.2.keytab.
I20260812 02:31:58.464978 14748 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbep4M4/build/debug/bin/kudu
/tmp/dist-test-taskbep4M4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786501898713699-14748-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786501898713699-14748-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786501898713699-14748-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786501898713699-14748-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.103.2:0
--local_ip_for_outbound_sockets=127.14.103.2
--webserver_interface=127.14.103.2
--webserver_port=0
--tserver_master_addrs=127.14.103.62:42887
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.2.keytab
--principal=kudu/127.14.103.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.103.20:46657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786501898713699-14748-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786501898713699-14748-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786501898713699-14748-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:31:58.601503 17655 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:31:58.601905 17655 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:31:58.602020 17655 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:31:58.606704 17655 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:31:58.606904 17655 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:31:58.607069 17655 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.103.2
I20260812 02:31:58.613268 17655 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.103.20:46657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786501898713699-14748-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786501898713699-14748-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.2.keytab
--principal=kudu/127.14.103.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.103.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786501898713699-14748-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.103.2
--webserver_port=0
--tserver_master_addrs=127.14.103.62:42887
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17655
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.103.2
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786501898713699-14748-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:31:58.614531 17655 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:31:58.615743 17655 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260812 02:31:58.629119 17655 server_base.cc:1034] running on GCE node
W20260812 02:31:58.629047 17663 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:31:58.629038 17660 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:31:58.629040 17661 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:31:58.629774 17655 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:31:58.630554 17655 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:31:58.633955 17655 hybrid_clock.cc:648] HybridClock initialized: now 1786501918633832 us; error 156 us; skew 500 ppm
Aug 12 02:31:58 dist-test-slave-2w3w krb5kdc[17381](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501918, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:31:58.637588 17655 init.cc:377] Logged in from keytab as kudu/127.14.103.2@KRBTEST.COM (short username kudu)
I20260812 02:31:58.638779 17655 webserver.cc:460] Webserver started at http://127.14.103.2:36363/ using document root <none> and password file <none>
I20260812 02:31:58.639550 17655 fs_manager.cc:359] Metadata directory not provided
I20260812 02:31:58.639751 17655 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:31:58.640090 17655 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:31:58.643812 17655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786501898713699-14748-0/minicluster-data/ts-1/data/instance:
uuid: "528e32e509fd4dc88c700217e8e88aef"
format_stamp: "Formatted at 2026-08-12 02:31:58 on dist-test-slave-2w3w"
server_key: "57aada80a1edc98388ee109a2e0156f3"
server_key_iv: "a2444e9b5415ac1685a3bb21ae3cfda2"
server_key_version: "encryptionkey@0"
I20260812 02:31:58.644642 17655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786501898713699-14748-0/minicluster-data/ts-1/wal/instance:
uuid: "528e32e509fd4dc88c700217e8e88aef"
format_stamp: "Formatted at 2026-08-12 02:31:58 on dist-test-slave-2w3w"
server_key: "57aada80a1edc98388ee109a2e0156f3"
server_key_iv: "a2444e9b5415ac1685a3bb21ae3cfda2"
server_key_version: "encryptionkey@0"
I20260812 02:31:58.650348 17655 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260812 02:31:58.653421 17670 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:31:58.655396 17655 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260812 02:31:58.655596 17655 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786501898713699-14748-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786501898713699-14748-0/minicluster-data/ts-1/wal
uuid: "528e32e509fd4dc88c700217e8e88aef"
format_stamp: "Formatted at 2026-08-12 02:31:58 on dist-test-slave-2w3w"
server_key: "57aada80a1edc98388ee109a2e0156f3"
server_key_iv: "a2444e9b5415ac1685a3bb21ae3cfda2"
server_key_version: "encryptionkey@0"
I20260812 02:31:58.655753 17655 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786501898713699-14748-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786501898713699-14748-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786501898713699-14748-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:31:58.669656 17655 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:31:58.674481 17655 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:31:58.674782 17655 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:31:58.675635 17655 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:31:58.676713 17655 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:31:58.676800 17655 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:31:58.676879 17655 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:31:58.676924 17655 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:31:58.692953 17655 rpc_server.cc:307] RPC server started. Bound to: 127.14.103.2:43851
I20260812 02:31:58.693074 17789 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.103.2:43851 every 8 connection(s)
I20260812 02:31:58.694042 17655 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786501898713699-14748-0/minicluster-data/ts-1/data/info.pb
I20260812 02:31:58.695757 14748 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbep4M4/build/debug/bin/kudu as pid 17655
I20260812 02:31:58.695878 14748 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786501898713699-14748-0/minicluster-data/ts-1/wal/instance
I20260812 02:31:58.696228 14748 external_mini_cluster.cc:1428] Setting key 7d80f0aa8bc7e3a9a2c43ab0042b7cd9
WARNING: no policy specified for kudu/127.14.103.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.103.3@KRBTEST.COM" created.
Aug 12 02:31:58 dist-test-slave-2w3w krb5kdc[17381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501918, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.2@KRBTEST.COM for kudu/127.14.103.62@KRBTEST.COM
I20260812 02:31:58.722537 17474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:31:58.696655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.2:51961 (local address 127.14.103.62:42887)
0812 02:31:58.696817 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0812 02:31:58.696894 (+ 77us) server_negotiation.cc:399] Waiting for connection header
0812 02:31:58.700428 (+ 3534us) server_negotiation.cc:407] Connection header received
0812 02:31:58.702535 (+ 2107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:31:58.702541 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:31:58.702652 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:31:58.702932 (+ 280us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:31:58.705394 (+ 2462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:31:58.706442 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:31:58.708179 (+ 1737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:31:58.708574 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:31:58.708731 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:31:58.716492 (+ 7761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:31:58.716498 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:31:58.716501 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:31:58.716566 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:31:58.718748 (+ 2182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:31:58.719801 (+ 1053us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:31:58.719808 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:31:58.719812 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:31:58.719938 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:31:58.720422 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:31:58.720429 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:31:58.720433 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:31:58.720981 (+ 548us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:31:58.721121 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0812 02:31:58.722097 (+ 976us) server_negotiation.cc:299] Negotiation successful
0812 02:31:58.722279 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260812 02:31:58.723357 17794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:31:58.698767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:42887 (local address 127.14.103.2:51961)
0812 02:31:58.700100 (+ 1333us) negotiation.cc:107] Waiting for socket to connect
0812 02:31:58.700229 (+ 129us) client_negotiation.cc:174] Beginning negotiation
0812 02:31:58.702048 (+ 1819us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:31:58.703269 (+ 1221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:31:58.703282 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:31:58.703966 (+ 684us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:31:58.704827 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:31:58.704856 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:31:58.706744 (+ 1888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:31:58.706748 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:31:58.707986 (+ 1238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:31:58.708001 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:31:58.708824 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:31:58.708828 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:31:58.709057 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:31:58.710250 (+ 1193us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:31:58.710275 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:31:58.715996 (+ 5721us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:31:58.719051 (+ 3055us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:31:58.719059 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:31:58.719076 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:31:58.719474 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:31:58.720065 (+ 591us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:31:58.720067 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:31:58.720069 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:31:58.720266 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:31:58.721140 (+ 874us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:31:58.721147 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:31:58.721719 (+ 572us) client_negotiation.cc:769] Sending connection context
0812 02:31:58.721914 (+ 195us) client_negotiation.cc:240] Negotiation successful
0812 02:31:58.722260 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":683,"thread_start_us":339,"threads_started":1}
I20260812 02:31:58.724993 17790 heartbeater.cc:347] Connected to a master server at 127.14.103.62:42887
I20260812 02:31:58.725323 17790 heartbeater.cc:464] Registering TS with master...
I20260812 02:31:58.726212 17790 heartbeater.cc:511] Master 127.14.103.62:42887 requested a full tablet report, sending...
I20260812 02:31:58.727773 17432 ts_manager.cc:194] Registered new tserver with Master: 528e32e509fd4dc88c700217e8e88aef (127.14.103.2:43851)
I20260812 02:31:58.728472 17432 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.103.2@KRBTEST.COM'} at 127.14.103.2:51961
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.103.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.3.keytab.
Entry for principal kudu/127.14.103.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.3.keytab.
I20260812 02:31:58.744699 14748 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbep4M4/build/debug/bin/kudu
/tmp/dist-test-taskbep4M4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786501898713699-14748-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786501898713699-14748-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786501898713699-14748-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786501898713699-14748-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.103.3:0
--local_ip_for_outbound_sockets=127.14.103.3
--webserver_interface=127.14.103.3
--webserver_port=0
--tserver_master_addrs=127.14.103.62:42887
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.3.keytab
--principal=kudu/127.14.103.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.103.20:46657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786501898713699-14748-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786501898713699-14748-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786501898713699-14748-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:31:58.891189 17797 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:31:58.891541 17797 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:31:58.891618 17797 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:31:58.896129 17797 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:31:58.896237 17797 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:31:58.896337 17797 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.103.3
I20260812 02:31:58.904587 17797 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.103.20:46657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786501898713699-14748-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786501898713699-14748-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.3.keytab
--principal=kudu/127.14.103.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.103.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786501898713699-14748-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.103.3
--webserver_port=0
--tserver_master_addrs=127.14.103.62:42887
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17797
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.103.3
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786501898713699-14748-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:31:58.906497 17797 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:31:58.907915 17797 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:31:58.920773 17807 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:31:58.920773 17804 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:31:58.920811 17805 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:31:58.921844 17797 server_base.cc:1034] running on GCE node
I20260812 02:31:58.922366 17797 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:31:58.923203 17797 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:31:58.926586 17797 hybrid_clock.cc:648] HybridClock initialized: now 1786501918926432 us; error 173 us; skew 500 ppm
Aug 12 02:31:58 dist-test-slave-2w3w krb5kdc[17381](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501918, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:31:58.930243 17797 init.cc:377] Logged in from keytab as kudu/127.14.103.3@KRBTEST.COM (short username kudu)
I20260812 02:31:58.931420 17797 webserver.cc:460] Webserver started at http://127.14.103.3:38275/ using document root <none> and password file <none>
I20260812 02:31:58.932273 17797 fs_manager.cc:359] Metadata directory not provided
I20260812 02:31:58.932374 17797 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:31:58.932632 17797 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:31:58.935621 17797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786501898713699-14748-0/minicluster-data/ts-2/data/instance:
uuid: "ea25e2080e5c4a01b03abc82e51d9125"
format_stamp: "Formatted at 2026-08-12 02:31:58 on dist-test-slave-2w3w"
server_key: "8e70656d04544193d853d90fcdb4d9b5"
server_key_iv: "5e3cdb8bac8d2176fc5e34a25813e1c6"
server_key_version: "encryptionkey@0"
I20260812 02:31:58.936550 17797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786501898713699-14748-0/minicluster-data/ts-2/wal/instance:
uuid: "ea25e2080e5c4a01b03abc82e51d9125"
format_stamp: "Formatted at 2026-08-12 02:31:58 on dist-test-slave-2w3w"
server_key: "8e70656d04544193d853d90fcdb4d9b5"
server_key_iv: "5e3cdb8bac8d2176fc5e34a25813e1c6"
server_key_version: "encryptionkey@0"
I20260812 02:31:58.941738 17797 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260812 02:31:58.945217 17814 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:31:58.946949 17797 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260812 02:31:58.947181 17797 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786501898713699-14748-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786501898713699-14748-0/minicluster-data/ts-2/wal
uuid: "ea25e2080e5c4a01b03abc82e51d9125"
format_stamp: "Formatted at 2026-08-12 02:31:58 on dist-test-slave-2w3w"
server_key: "8e70656d04544193d853d90fcdb4d9b5"
server_key_iv: "5e3cdb8bac8d2176fc5e34a25813e1c6"
server_key_version: "encryptionkey@0"
I20260812 02:31:58.947299 17797 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786501898713699-14748-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786501898713699-14748-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786501898713699-14748-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:31:58.971892 17797 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:31:58.976425 17797 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:31:58.976779 17797 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:31:58.977706 17797 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:31:58.978821 17797 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:31:58.978904 17797 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:31:58.979031 17797 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:31:58.979086 17797 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260812 02:31:58.990909 17797 rpc_server.cc:307] RPC server started. Bound to: 127.14.103.3:46239
I20260812 02:31:58.990916 17934 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.103.3:46239 every 8 connection(s)
I20260812 02:31:58.992173 17797 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786501898713699-14748-0/minicluster-data/ts-2/data/info.pb
I20260812 02:31:58.997331 14748 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbep4M4/build/debug/bin/kudu as pid 17797
I20260812 02:31:58.997493 14748 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786501898713699-14748-0/minicluster-data/ts-2/wal/instance
I20260812 02:31:58.997859 14748 external_mini_cluster.cc:1428] Setting key a45a4f472e7e6bb9f279f325e79ef39f
Aug 12 02:31:59 dist-test-slave-2w3w krb5kdc[17381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501918, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.3@KRBTEST.COM for kudu/127.14.103.62@KRBTEST.COM
I20260812 02:31:59.012727 17474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:31:58.994479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.3:60475 (local address 127.14.103.62:42887)
0812 02:31:58.994651 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0812 02:31:58.994656 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:31:58.995758 (+ 1102us) server_negotiation.cc:407] Connection header received
0812 02:31:58.997784 (+ 2026us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:31:58.997787 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:31:58.997850 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:31:58.998054 (+ 204us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:31:58.999907 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:31:59.001341 (+ 1434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:31:59.003114 (+ 1773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:31:59.003471 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:31:59.003592 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:31:59.007355 (+ 3763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:31:59.007359 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:31:59.007361 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:31:59.007397 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:31:59.009154 (+ 1757us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:31:59.010052 (+ 898us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:31:59.010057 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:31:59.010059 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:31:59.010479 (+ 420us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:31:59.011116 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:31:59.011120 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:31:59.011121 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:31:59.011508 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:31:59.011635 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0812 02:31:59.012373 (+ 738us) server_negotiation.cc:299] Negotiation successful
0812 02:31:59.012547 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260812 02:31:59.013757 17937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:31:58.994943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:42887 (local address 127.14.103.3:60475)
0812 02:31:58.995532 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0812 02:31:58.995582 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0812 02:31:58.997430 (+ 1848us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:31:58.998290 (+ 860us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:31:58.998310 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:31:58.998807 (+ 497us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:31:58.999615 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:31:58.999643 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:31:59.001601 (+ 1958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:31:59.001604 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:31:59.002750 (+ 1146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:31:59.002764 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:31:59.003714 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:31:59.003718 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:31:59.003885 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:31:59.004720 (+ 835us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:31:59.004738 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:31:59.006769 (+ 2031us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:31:59.009368 (+ 2599us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:31:59.009376 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:31:59.009394 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:31:59.009754 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:31:59.010647 (+ 893us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:31:59.010650 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:31:59.010651 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:31:59.010872 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:31:59.011691 (+ 819us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:31:59.011699 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:31:59.012073 (+ 374us) client_negotiation.cc:769] Sending connection context
0812 02:31:59.012304 (+ 231us) client_negotiation.cc:240] Negotiation successful
0812 02:31:59.012712 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":143,"threads_started":1}
I20260812 02:31:59.015556 17935 heartbeater.cc:347] Connected to a master server at 127.14.103.62:42887
I20260812 02:31:59.015826 17935 heartbeater.cc:464] Registering TS with master...
I20260812 02:31:59.016393 17935 heartbeater.cc:511] Master 127.14.103.62:42887 requested a full tablet report, sending...
I20260812 02:31:59.017813 17432 ts_manager.cc:194] Registered new tserver with Master: ea25e2080e5c4a01b03abc82e51d9125 (127.14.103.3:46239)
I20260812 02:31:59.018613 17432 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.103.3@KRBTEST.COM'} at 127.14.103.3:60475
I20260812 02:31:59.026659 14748 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260812 02:31:59.045217 17474 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:31:59.043203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49206 (local address 127.14.103.62:42887)
0812 02:31:59.043359 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0812 02:31:59.043364 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:31:59.043562 (+ 198us) server_negotiation.cc:407] Connection header received
0812 02:31:59.043687 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:31:59.043690 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:31:59.043744 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:31:59.043846 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:31:59.045003 (+ 1157us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49206: BlockingRecv error: recv got EOF from 127.0.0.1:49206 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
I20260812 02:31:59.047443 14748 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbep4M4/build/debug/bin/kudu with pid 17502
I20260812 02:31:59.057677 14748 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbep4M4/build/debug/bin/kudu with pid 17655
I20260812 02:31:59.068608 14748 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbep4M4/build/debug/bin/kudu with pid 17797
I20260812 02:31:59.078924 14748 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbep4M4/build/debug/bin/kudu with pid 17397
2026-08-12T02:31:59Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3396 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786501898713699-14748-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 02:31:59 dist-test-slave-2w3w krb5kdc[17950](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 02:31:59 dist-test-slave-2w3w krb5kdc[17950](info): set up 2 sockets
Aug 12 02:31:59 dist-test-slave-2w3w krb5kdc[17950](info): commencing operation
krb5kdc: starting...
W20260812 02:32:00.254215 14748 mini_kdc.cc:121] Time spent starting KDC: real 1.146s user 0.003s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786501898713699-14748-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786501898713699-14748-0/krb5kdc/test-user.keytab.
Aug 12 02:32:00 dist-test-slave-2w3w krb5kdc[17950](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501920, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T02:32:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T02:32:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.103.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.103.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.103.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.62.keytab.
Entry for principal kudu/127.14.103.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.62.keytab.
I20260812 02:32:00.403363 14748 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbep4M4/build/debug/bin/kudu
/tmp/dist-test-taskbep4M4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786501898713699-14748-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786501898713699-14748-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786501898713699-14748-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786501898713699-14748-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.103.62:36599
--webserver_interface=127.14.103.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.62.keytab
--principal=kudu/127.14.103.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.103.20:43041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.103.62:36599
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786501898713699-14748-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786501898713699-14748-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786501898713699-14748-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:32:00.528937 17963 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:32:00.529328 17963 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:32:00.529417 17963 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:32:00.533900 17963 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:32:00.534029 17963 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:32:00.534070 17963 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:32:00.534111 17963 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:32:00.534149 17963 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 02:32:00.540560 17963 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.103.20:43041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786501898713699-14748-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786501898713699-14748-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.103.62:36599
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.62.keytab
--principal=kudu/127.14.103.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.103.62:36599
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786501898713699-14748-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.103.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17963
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786501898713699-14748-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:32:00.541826 17963 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:32:00.542937 17963 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:32:00.552098 17969 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:32:00.552099 17972 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:32:00.552454 17963 server_base.cc:1034] running on GCE node
W20260812 02:32:00.552142 17968 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:32:00.553105 17963 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:32:00.554358 17963 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:32:00.555542 17963 hybrid_clock.cc:648] HybridClock initialized: now 1786501920555511 us; error 46 us; skew 500 ppm
Aug 12 02:32:00 dist-test-slave-2w3w krb5kdc[17950](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501920, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:32:00.558687 17963 init.cc:377] Logged in from keytab as kudu/127.14.103.62@KRBTEST.COM (short username kudu)
I20260812 02:32:00.559752 17963 webserver.cc:460] Webserver started at http://127.14.103.62:40151/ using document root <none> and password file <none>
I20260812 02:32:00.560352 17963 fs_manager.cc:359] Metadata directory not provided
I20260812 02:32:00.560420 17963 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:32:00.560642 17963 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:32:00.562881 17963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786501898713699-14748-0/minicluster-data/master-0/data/instance:
uuid: "6844b347f8b44f6eba9c063656eaa2fb"
format_stamp: "Formatted at 2026-08-12 02:32:00 on dist-test-slave-2w3w"
server_key: "47e67d73ceec5aceef87c78044bf9857"
server_key_iv: "fbce41f00fcb94d1a36328e0fde979df"
server_key_version: "encryptionkey@0"
I20260812 02:32:00.563613 17963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786501898713699-14748-0/minicluster-data/master-0/wal/instance:
uuid: "6844b347f8b44f6eba9c063656eaa2fb"
format_stamp: "Formatted at 2026-08-12 02:32:00 on dist-test-slave-2w3w"
server_key: "47e67d73ceec5aceef87c78044bf9857"
server_key_iv: "fbce41f00fcb94d1a36328e0fde979df"
server_key_version: "encryptionkey@0"
I20260812 02:32:00.567759 17963 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260812 02:32:00.570569 17979 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:32:00.572072 17963 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260812 02:32:00.572249 17963 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786501898713699-14748-0/minicluster-data/master-0/data,/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786501898713699-14748-0/minicluster-data/master-0/wal
uuid: "6844b347f8b44f6eba9c063656eaa2fb"
format_stamp: "Formatted at 2026-08-12 02:32:00 on dist-test-slave-2w3w"
server_key: "47e67d73ceec5aceef87c78044bf9857"
server_key_iv: "fbce41f00fcb94d1a36328e0fde979df"
server_key_version: "encryptionkey@0"
I20260812 02:32:00.572423 17963 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786501898713699-14748-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786501898713699-14748-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786501898713699-14748-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:32:00.600837 17963 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:32:00.605646 17963 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:32:00.605899 17963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:32:00.614504 17963 rpc_server.cc:307] RPC server started. Bound to: 127.14.103.62:36599
I20260812 02:32:00.614936 18034 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.103.62:36599 every 8 connection(s)
I20260812 02:32:00.615696 17963 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786501898713699-14748-0/minicluster-data/master-0/data/info.pb
I20260812 02:32:00.619114 18035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:32:00.622054 14748 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbep4M4/build/debug/bin/kudu as pid 17963
I20260812 02:32:00.622224 14748 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786501898713699-14748-0/minicluster-data/master-0/wal/instance
I20260812 02:32:00.622514 14748 external_mini_cluster.cc:1428] Setting key 6dcc5759e4c670e4c5adedaa6e95b27d
I20260812 02:32:00.625870 18035 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6844b347f8b44f6eba9c063656eaa2fb: Bootstrap starting.
I20260812 02:32:00.628887 18035 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6844b347f8b44f6eba9c063656eaa2fb: Neither blocks nor log segments found. Creating new log.
I20260812 02:32:00.630105 18035 log.cc:824] T 00000000000000000000000000000000 P 6844b347f8b44f6eba9c063656eaa2fb: Log is configured to *not* fsync() on all Append() calls
I20260812 02:32:00.632527 18035 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6844b347f8b44f6eba9c063656eaa2fb: No bootstrap required, opened a new log
Aug 12 02:32:00 dist-test-slave-2w3w krb5kdc[17950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501920, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.103.62@KRBTEST.COM
I20260812 02:32:00.637216 18035 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6844b347f8b44f6eba9c063656eaa2fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6844b347f8b44f6eba9c063656eaa2fb" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 36599 } }
I20260812 02:32:00.637578 18035 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6844b347f8b44f6eba9c063656eaa2fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:32:00.637645 18035 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6844b347f8b44f6eba9c063656eaa2fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6844b347f8b44f6eba9c063656eaa2fb, State: Initialized, Role: FOLLOWER
I20260812 02:32:00.638177 18035 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6844b347f8b44f6eba9c063656eaa2fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6844b347f8b44f6eba9c063656eaa2fb" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 36599 } }
I20260812 02:32:00.638316 18035 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6844b347f8b44f6eba9c063656eaa2fb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 02:32:00.638413 18035 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6844b347f8b44f6eba9c063656eaa2fb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 02:32:00.638520 18035 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6844b347f8b44f6eba9c063656eaa2fb [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:32:00.640372 18039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:00.624307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49278 (local address 127.14.103.62:36599)
0812 02:32:00.624810 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:00.624826 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:00.624864 (+ 38us) server_negotiation.cc:407] Connection header received
0812 02:32:00.625509 (+ 645us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:00.625526 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:00.625892 (+ 366us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:00.626253 (+ 361us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:32:00.627357 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:00.628606 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:00.630123 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:00.630812 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:00.631018 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:00.633665 (+ 2647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:32:00.633672 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:32:00.633690 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:32:00.633726 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:32:00.636773 (+ 3047us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:00.637382 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:00.637389 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:00.637395 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:00.637551 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:00.637959 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:00.637962 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:00.637964 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:00.638597 (+ 633us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:32:00.638833 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0812 02:32:00.639193 (+ 360us) server_negotiation.cc:299] Negotiation successful
0812 02:32:00.639554 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":169,"threads_started":1}
I20260812 02:32:00.640333 18035 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6844b347f8b44f6eba9c063656eaa2fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6844b347f8b44f6eba9c063656eaa2fb" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 36599 } }
I20260812 02:32:00.640925 18035 leader_election.cc:302] T 00000000000000000000000000000000 P 6844b347f8b44f6eba9c063656eaa2fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6844b347f8b44f6eba9c063656eaa2fb; no voters:
I20260812 02:32:00.641314 18035 leader_election.cc:288] T 00000000000000000000000000000000 P 6844b347f8b44f6eba9c063656eaa2fb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 02:32:00.641454 18041 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6844b347f8b44f6eba9c063656eaa2fb [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:32:00.642419 18041 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6844b347f8b44f6eba9c063656eaa2fb [term 1 LEADER]: Becoming Leader. State: Replica: 6844b347f8b44f6eba9c063656eaa2fb, State: Running, Role: LEADER
I20260812 02:32:00.642763 18035 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6844b347f8b44f6eba9c063656eaa2fb [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:32:00.642758 18041 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6844b347f8b44f6eba9c063656eaa2fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6844b347f8b44f6eba9c063656eaa2fb" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 36599 } }
I20260812 02:32:00.643970 14748 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 02:32:00.644773 18042 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6844b347f8b44f6eba9c063656eaa2fb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6844b347f8b44f6eba9c063656eaa2fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6844b347f8b44f6eba9c063656eaa2fb" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 36599 } } }
I20260812 02:32:00.644781 18043 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6844b347f8b44f6eba9c063656eaa2fb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6844b347f8b44f6eba9c063656eaa2fb. Latest consensus state: current_term: 1 leader_uuid: "6844b347f8b44f6eba9c063656eaa2fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6844b347f8b44f6eba9c063656eaa2fb" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 36599 } } }
I20260812 02:32:00.644943 18042 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6844b347f8b44f6eba9c063656eaa2fb [sys.catalog]: This master's current role is: LEADER
I20260812 02:32:00.644949 18043 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6844b347f8b44f6eba9c063656eaa2fb [sys.catalog]: This master's current role is: LEADER
I20260812 02:32:00.645339 18057 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 02:32:00.649065 18057 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 02:32:00.656796 18057 catalog_manager.cc:1347] Generated new cluster ID: e2c25061520642368b0156dcd78a3eeb
I20260812 02:32:00.657001 18057 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 02:32:00.669900 18039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:00.661669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49294 (local address 127.14.103.62:36599)
0812 02:32:00.661821 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:00.661827 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:00.662225 (+ 398us) server_negotiation.cc:407] Connection header received
0812 02:32:00.662390 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:00.662393 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:00.662462 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:00.662558 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:32:00.664761 (+ 2203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:00.665587 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:00.667300 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:00.667599 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:00.667771 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:00.668781 (+ 1010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:32:00.668785 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:32:00.668788 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:32:00.669323 (+ 535us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:32:00.669358 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:32:00.669376 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:32:00.669498 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0812 02:32:00.669609 (+ 111us) server_negotiation.cc:299] Negotiation successful
0812 02:32:00.669691 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":47}
I20260812 02:32:00.674180 18057 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 02:32:00.675527 18057 catalog_manager.cc:1504] Loading token signing keys...
I20260812 02:32:00.688464 18057 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6844b347f8b44f6eba9c063656eaa2fb: Generated new TSK 0
I20260812 02:32:00.689378 18057 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 02:32:00.698161 14748 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbep4M4/build/debug/bin/kudu with pid 17963
2026-08-12T02:32:00Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1616 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786501898713699-14748-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 02:32:00 dist-test-slave-2w3w krb5kdc[18072](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 02:32:00 dist-test-slave-2w3w krb5kdc[18072](info): set up 2 sockets
Aug 12 02:32:00 dist-test-slave-2w3w krb5kdc[18072](info): commencing operation
krb5kdc: starting...
W20260812 02:32:01.871483 14748 mini_kdc.cc:121] Time spent starting KDC: real 1.147s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786501898713699-14748-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786501898713699-14748-0/krb5kdc/test-user.keytab.
Aug 12 02:32:01 dist-test-slave-2w3w krb5kdc[18072](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501921, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T02:32:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T02:32:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.103.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.103.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.103.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.62.keytab.
Entry for principal kudu/127.14.103.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.62.keytab.
I20260812 02:32:02.027606 14748 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbep4M4/build/debug/bin/kudu
/tmp/dist-test-taskbep4M4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786501898713699-14748-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786501898713699-14748-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786501898713699-14748-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786501898713699-14748-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.103.62:39073
--webserver_interface=127.14.103.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.62.keytab
--principal=kudu/127.14.103.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.103.20:37097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.103.62:39073
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786501898713699-14748-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786501898713699-14748-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786501898713699-14748-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:32:02.164875 18089 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:32:02.165190 18089 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:32:02.165266 18089 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:32:02.170769 18089 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:32:02.170889 18089 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:32:02.170912 18089 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:32:02.170941 18089 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:32:02.171029 18089 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 02:32:02.177498 18089 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.103.20:37097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786501898713699-14748-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786501898713699-14748-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.103.62:39073
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.62.keytab
--principal=kudu/127.14.103.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.103.62:39073
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786501898713699-14748-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.103.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18089
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786501898713699-14748-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:32:02.179399 18089 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:32:02.182489 18089 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:32:02.195470 18094 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:32:02.195659 18089 server_base.cc:1034] running on GCE node
W20260812 02:32:02.195508 18096 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:32:02.195760 18098 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:32:02.196503 18089 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:32:02.197824 18089 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:32:02.199229 18089 hybrid_clock.cc:648] HybridClock initialized: now 1786501922199149 us; error 97 us; skew 500 ppm
Aug 12 02:32:02 dist-test-slave-2w3w krb5kdc[18072](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501922, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:32:02.204180 18089 init.cc:377] Logged in from keytab as kudu/127.14.103.62@KRBTEST.COM (short username kudu)
I20260812 02:32:02.205570 18089 webserver.cc:460] Webserver started at http://127.14.103.62:34411/ using document root <none> and password file <none>
I20260812 02:32:02.206243 18089 fs_manager.cc:359] Metadata directory not provided
I20260812 02:32:02.206314 18089 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:32:02.206513 18089 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:32:02.208899 18089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786501898713699-14748-0/minicluster-data/master-0/data/instance:
uuid: "a73279ed8d69464da704a80c40faa595"
format_stamp: "Formatted at 2026-08-12 02:32:02 on dist-test-slave-2w3w"
server_key: "45c4eaeb2f20908f701fe294e1071663"
server_key_iv: "117a1fd0fc5c926cb35a679aaafe03f3"
server_key_version: "encryptionkey@0"
I20260812 02:32:02.209568 18089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786501898713699-14748-0/minicluster-data/master-0/wal/instance:
uuid: "a73279ed8d69464da704a80c40faa595"
format_stamp: "Formatted at 2026-08-12 02:32:02 on dist-test-slave-2w3w"
server_key: "45c4eaeb2f20908f701fe294e1071663"
server_key_iv: "117a1fd0fc5c926cb35a679aaafe03f3"
server_key_version: "encryptionkey@0"
I20260812 02:32:02.214021 18089 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260812 02:32:02.218053 18105 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:32:02.220113 18089 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.002s
I20260812 02:32:02.220299 18089 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786501898713699-14748-0/minicluster-data/master-0/data,/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786501898713699-14748-0/minicluster-data/master-0/wal
uuid: "a73279ed8d69464da704a80c40faa595"
format_stamp: "Formatted at 2026-08-12 02:32:02 on dist-test-slave-2w3w"
server_key: "45c4eaeb2f20908f701fe294e1071663"
server_key_iv: "117a1fd0fc5c926cb35a679aaafe03f3"
server_key_version: "encryptionkey@0"
I20260812 02:32:02.220440 18089 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786501898713699-14748-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786501898713699-14748-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786501898713699-14748-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:32:02.239496 18089 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:32:02.243245 18089 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:32:02.243435 18089 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:32:02.253118 18089 rpc_server.cc:307] RPC server started. Bound to: 127.14.103.62:39073
I20260812 02:32:02.253603 18163 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.103.62:39073 every 8 connection(s)
I20260812 02:32:02.254858 18089 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786501898713699-14748-0/minicluster-data/master-0/data/info.pb
I20260812 02:32:02.256788 14748 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbep4M4/build/debug/bin/kudu as pid 18089
I20260812 02:32:02.256942 14748 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786501898713699-14748-0/minicluster-data/master-0/wal/instance
I20260812 02:32:02.257261 14748 external_mini_cluster.cc:1428] Setting key 6feec0c1050abaa55a35c8becb2d3c49
I20260812 02:32:02.259451 18164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:32:02.268224 18164 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a73279ed8d69464da704a80c40faa595: Bootstrap starting.
Aug 12 02:32:02 dist-test-slave-2w3w krb5kdc[18072](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501921, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.103.62@KRBTEST.COM
I20260812 02:32:02.271320 18164 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a73279ed8d69464da704a80c40faa595: Neither blocks nor log segments found. Creating new log.
I20260812 02:32:02.272338 18164 log.cc:824] T 00000000000000000000000000000000 P a73279ed8d69464da704a80c40faa595: Log is configured to *not* fsync() on all Append() calls
I20260812 02:32:02.275204 18164 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a73279ed8d69464da704a80c40faa595: No bootstrap required, opened a new log
I20260812 02:32:02.276012 18167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:02.259354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45368 (local address 127.14.103.62:39073)
0812 02:32:02.259968 (+ 614us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:02.259993 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:02.260061 (+ 68us) server_negotiation.cc:407] Connection header received
0812 02:32:02.261211 (+ 1150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:02.261238 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:02.261748 (+ 510us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:02.262175 (+ 427us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:32:02.263188 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:02.264285 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:02.265647 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:02.266205 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:02.266376 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:02.269759 (+ 3383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:32:02.269768 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:32:02.269787 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:32:02.269830 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:32:02.272208 (+ 2378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:02.272856 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:02.272863 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:02.272869 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:02.273025 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:02.273423 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:02.273426 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:02.273428 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:02.273991 (+ 563us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:32:02.274203 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0812 02:32:02.274491 (+ 288us) server_negotiation.cc:299] Negotiation successful
0812 02:32:02.274846 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":175,"threads_started":1}
I20260812 02:32:02.279659 18164 raft_consensus.cc:348] T 00000000000000000000000000000000 P a73279ed8d69464da704a80c40faa595 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a73279ed8d69464da704a80c40faa595" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 39073 } }
I20260812 02:32:02.279922 18164 raft_consensus.cc:374] T 00000000000000000000000000000000 P a73279ed8d69464da704a80c40faa595 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:32:02.279960 18164 raft_consensus.cc:729] T 00000000000000000000000000000000 P a73279ed8d69464da704a80c40faa595 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a73279ed8d69464da704a80c40faa595, State: Initialized, Role: FOLLOWER
I20260812 02:32:02.280454 18164 consensus_queue.cc:260] T 00000000000000000000000000000000 P a73279ed8d69464da704a80c40faa595 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a73279ed8d69464da704a80c40faa595" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 39073 } }
I20260812 02:32:02.280658 18164 raft_consensus.cc:388] T 00000000000000000000000000000000 P a73279ed8d69464da704a80c40faa595 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 02:32:02.280746 18164 raft_consensus.cc:482] T 00000000000000000000000000000000 P a73279ed8d69464da704a80c40faa595 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 02:32:02.280874 18164 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a73279ed8d69464da704a80c40faa595 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:32:02.281986 18164 raft_consensus.cc:504] T 00000000000000000000000000000000 P a73279ed8d69464da704a80c40faa595 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a73279ed8d69464da704a80c40faa595" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 39073 } }
I20260812 02:32:02.282397 18164 leader_election.cc:302] T 00000000000000000000000000000000 P a73279ed8d69464da704a80c40faa595 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a73279ed8d69464da704a80c40faa595; no voters:
I20260812 02:32:02.282804 18164 leader_election.cc:288] T 00000000000000000000000000000000 P a73279ed8d69464da704a80c40faa595 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 02:32:02.282912 18170 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a73279ed8d69464da704a80c40faa595 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:32:02.283107 18170 raft_consensus.cc:686] T 00000000000000000000000000000000 P a73279ed8d69464da704a80c40faa595 [term 1 LEADER]: Becoming Leader. State: Replica: a73279ed8d69464da704a80c40faa595, State: Running, Role: LEADER
I20260812 02:32:02.283438 18170 consensus_queue.cc:237] T 00000000000000000000000000000000 P a73279ed8d69464da704a80c40faa595 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a73279ed8d69464da704a80c40faa595" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 39073 } }
I20260812 02:32:02.284282 18164 sys_catalog.cc:564] T 00000000000000000000000000000000 P a73279ed8d69464da704a80c40faa595 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:32:02.285307 18171 sys_catalog.cc:455] T 00000000000000000000000000000000 P a73279ed8d69464da704a80c40faa595 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a73279ed8d69464da704a80c40faa595. Latest consensus state: current_term: 1 leader_uuid: "a73279ed8d69464da704a80c40faa595" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a73279ed8d69464da704a80c40faa595" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 39073 } } }
I20260812 02:32:02.285456 18171 sys_catalog.cc:458] T 00000000000000000000000000000000 P a73279ed8d69464da704a80c40faa595 [sys.catalog]: This master's current role is: LEADER
I20260812 02:32:02.285753 18172 sys_catalog.cc:455] T 00000000000000000000000000000000 P a73279ed8d69464da704a80c40faa595 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a73279ed8d69464da704a80c40faa595" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a73279ed8d69464da704a80c40faa595" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 39073 } } }
I20260812 02:32:02.285848 18172 sys_catalog.cc:458] T 00000000000000000000000000000000 P a73279ed8d69464da704a80c40faa595 [sys.catalog]: This master's current role is: LEADER
I20260812 02:32:02.285923 18180 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 02:32:02.289393 18180 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 02:32:02.295945 18180 catalog_manager.cc:1347] Generated new cluster ID: dc6f7875a7124d889f53df90549d2ab8
I20260812 02:32:02.296049 18180 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 02:32:02.325354 18180 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 02:32:02.326485 18180 catalog_manager.cc:1504] Loading token signing keys...
I20260812 02:32:02.331086 18180 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a73279ed8d69464da704a80c40faa595: Generated new TSK 0
I20260812 02:32:02.332036 18180 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.103.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.103.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.103.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.1.keytab.
Entry for principal kudu/127.14.103.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.1.keytab.
I20260812 02:32:02.383966 14748 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbep4M4/build/debug/bin/kudu
/tmp/dist-test-taskbep4M4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786501898713699-14748-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786501898713699-14748-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786501898713699-14748-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786501898713699-14748-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.103.1:0
--local_ip_for_outbound_sockets=127.14.103.1
--webserver_interface=127.14.103.1
--webserver_port=0
--tserver_master_addrs=127.14.103.62:39073
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.1.keytab
--principal=kudu/127.14.103.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.103.20:37097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786501898713699-14748-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786501898713699-14748-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786501898713699-14748-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:32:02.514441 18193 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:32:02.514806 18193 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:32:02.514866 18193 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:32:02.519940 18193 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:32:02.520047 18193 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:32:02.520121 18193 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.103.1
I20260812 02:32:02.527459 18193 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.103.20:37097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786501898713699-14748-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786501898713699-14748-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.1.keytab
--principal=kudu/127.14.103.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.103.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786501898713699-14748-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.103.1
--webserver_port=0
--tserver_master_addrs=127.14.103.62:39073
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18193
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.103.1
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786501898713699-14748-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:32:02.528672 18193 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:32:02.529837 18193 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:32:02.542128 18201 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:32:02.542718 18199 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:32:02.542945 18193 server_base.cc:1034] running on GCE node
W20260812 02:32:02.543308 18198 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:32:02.544437 18193 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:32:02.545369 18193 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:32:02.546639 18193 hybrid_clock.cc:648] HybridClock initialized: now 1786501922546591 us; error 83 us; skew 500 ppm
Aug 12 02:32:02 dist-test-slave-2w3w krb5kdc[18072](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501922, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:32:02.550194 18193 init.cc:377] Logged in from keytab as kudu/127.14.103.1@KRBTEST.COM (short username kudu)
I20260812 02:32:02.551462 18193 webserver.cc:460] Webserver started at http://127.14.103.1:43451/ using document root <none> and password file <none>
I20260812 02:32:02.552098 18193 fs_manager.cc:359] Metadata directory not provided
I20260812 02:32:02.552209 18193 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:32:02.552476 18193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:32:02.555302 18193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786501898713699-14748-0/minicluster-data/ts-0/data/instance:
uuid: "55f99e809d3644379b2b02fc54073818"
format_stamp: "Formatted at 2026-08-12 02:32:02 on dist-test-slave-2w3w"
server_key: "cf0f0f1fbafd8e4fe3c9182d5634447c"
server_key_iv: "b150d5275672762adcaa053db7aab7b7"
server_key_version: "encryptionkey@0"
I20260812 02:32:02.556035 18193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786501898713699-14748-0/minicluster-data/ts-0/wal/instance:
uuid: "55f99e809d3644379b2b02fc54073818"
format_stamp: "Formatted at 2026-08-12 02:32:02 on dist-test-slave-2w3w"
server_key: "cf0f0f1fbafd8e4fe3c9182d5634447c"
server_key_iv: "b150d5275672762adcaa053db7aab7b7"
server_key_version: "encryptionkey@0"
I20260812 02:32:02.561893 18193 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260812 02:32:02.565789 18208 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:32:02.567519 18193 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260812 02:32:02.567709 18193 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786501898713699-14748-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786501898713699-14748-0/minicluster-data/ts-0/wal
uuid: "55f99e809d3644379b2b02fc54073818"
format_stamp: "Formatted at 2026-08-12 02:32:02 on dist-test-slave-2w3w"
server_key: "cf0f0f1fbafd8e4fe3c9182d5634447c"
server_key_iv: "b150d5275672762adcaa053db7aab7b7"
server_key_version: "encryptionkey@0"
I20260812 02:32:02.567870 18193 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786501898713699-14748-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786501898713699-14748-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786501898713699-14748-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:32:02.584928 18193 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:32:02.589455 18193 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:32:02.589725 18193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:32:02.590417 18193 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:32:02.592106 18193 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:32:02.592263 18193 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:32:02.592392 18193 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:32:02.592433 18193 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:32:02.603183 18193 rpc_server.cc:307] RPC server started. Bound to: 127.14.103.1:34015
I20260812 02:32:02.603505 18334 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.103.1:34015 every 8 connection(s)
I20260812 02:32:02.604543 18193 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786501898713699-14748-0/minicluster-data/ts-0/data/info.pb
I20260812 02:32:02.614420 14748 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbep4M4/build/debug/bin/kudu as pid 18193
I20260812 02:32:02.614542 14748 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786501898713699-14748-0/minicluster-data/ts-0/wal/instance
I20260812 02:32:02.615020 14748 external_mini_cluster.cc:1428] Setting key e525253590d7a465c9e332077c1e6e56
Aug 12 02:32:02 dist-test-slave-2w3w krb5kdc[18072](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501922, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.1@KRBTEST.COM for kudu/127.14.103.62@KRBTEST.COM
I20260812 02:32:02.623567 18167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:02.607663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.1:37345 (local address 127.14.103.62:39073)
0812 02:32:02.607855 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:02.607859 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:02.608822 (+ 963us) server_negotiation.cc:407] Connection header received
0812 02:32:02.610334 (+ 1512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:02.610337 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:02.610405 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:02.610514 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:32:02.611993 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:02.612559 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:02.613909 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:02.614187 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:02.614321 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:02.617807 (+ 3486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:32:02.617815 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:32:02.617821 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:32:02.617863 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:32:02.619997 (+ 2134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:02.621064 (+ 1067us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:02.621072 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:02.621078 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:02.621211 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:02.621674 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:02.621681 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:02.621686 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:02.622100 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:32:02.622245 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0812 02:32:02.623037 (+ 792us) server_negotiation.cc:299] Negotiation successful
0812 02:32:02.623266 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260812 02:32:02.625108 18337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:02.607941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:39073 (local address 127.14.103.1:37345)
0812 02:32:02.608625 (+ 684us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:02.608677 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0812 02:32:02.610068 (+ 1391us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:32:02.610727 (+ 659us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:32:02.610735 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:32:02.611267 (+ 532us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:32:02.611825 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:02.611840 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:02.612713 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:02.612717 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:02.613720 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:02.613731 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:02.614368 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:02.614372 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:02.614612 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:02.615385 (+ 773us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:32:02.615403 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:32:02.617520 (+ 2117us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:32:02.620281 (+ 2761us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:32:02.620294 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:32:02.620327 (+ 33us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:32:02.620856 (+ 529us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:32:02.621346 (+ 490us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:32:02.621348 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:32:02.621350 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:32:02.621540 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:32:02.622344 (+ 804us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:32:02.622352 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:32:02.622667 (+ 315us) client_negotiation.cc:769] Sending connection context
0812 02:32:02.622885 (+ 218us) client_negotiation.cc:240] Negotiation successful
0812 02:32:02.623954 (+ 1069us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":174,"threads_started":1}
I20260812 02:32:02.627169 18335 heartbeater.cc:347] Connected to a master server at 127.14.103.62:39073
I20260812 02:32:02.627621 18335 heartbeater.cc:464] Registering TS with master...
I20260812 02:32:02.628470 18335 heartbeater.cc:511] Master 127.14.103.62:39073 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.14.103.2@KRBTEST.COM; defaulting to no policy
I20260812 02:32:02.630532 18123 ts_manager.cc:194] Registered new tserver with Master: 55f99e809d3644379b2b02fc54073818 (127.14.103.1:34015)
I20260812 02:32:02.631887 18123 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.103.1@KRBTEST.COM'} at 127.14.103.1:37345
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.103.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.103.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.2.keytab.
Entry for principal kudu/127.14.103.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.2.keytab.
I20260812 02:32:02.656769 14748 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbep4M4/build/debug/bin/kudu
/tmp/dist-test-taskbep4M4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786501898713699-14748-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786501898713699-14748-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786501898713699-14748-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786501898713699-14748-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.103.2:0
--local_ip_for_outbound_sockets=127.14.103.2
--webserver_interface=127.14.103.2
--webserver_port=0
--tserver_master_addrs=127.14.103.62:39073
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.2.keytab
--principal=kudu/127.14.103.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.103.20:37097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786501898713699-14748-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786501898713699-14748-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786501898713699-14748-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:32:02.791060 18340 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:32:02.791555 18340 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:32:02.791687 18340 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:32:02.796310 18340 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:32:02.796387 18340 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:32:02.796453 18340 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.103.2
I20260812 02:32:02.803062 18340 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.103.20:37097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786501898713699-14748-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786501898713699-14748-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.2.keytab
--principal=kudu/127.14.103.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.103.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786501898713699-14748-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.103.2
--webserver_port=0
--tserver_master_addrs=127.14.103.62:39073
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18340
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.103.2
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786501898713699-14748-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:32:02.804256 18340 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:32:02.805472 18340 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:32:02.817106 18346 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:32:02.817304 18349 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:32:02.817106 18347 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:32:02.817682 18340 server_base.cc:1034] running on GCE node
I20260812 02:32:02.818173 18340 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:32:02.818912 18340 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:32:02.820165 18340 hybrid_clock.cc:648] HybridClock initialized: now 1786501922820123 us; error 57 us; skew 500 ppm
Aug 12 02:32:02 dist-test-slave-2w3w krb5kdc[18072](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501922, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:32:02.823786 18340 init.cc:377] Logged in from keytab as kudu/127.14.103.2@KRBTEST.COM (short username kudu)
I20260812 02:32:02.825277 18340 webserver.cc:460] Webserver started at http://127.14.103.2:38095/ using document root <none> and password file <none>
I20260812 02:32:02.826022 18340 fs_manager.cc:359] Metadata directory not provided
I20260812 02:32:02.826136 18340 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:32:02.826423 18340 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:32:02.828979 18340 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786501898713699-14748-0/minicluster-data/ts-1/data/instance:
uuid: "93a0e7bd61b646869872c5626a0ea41d"
format_stamp: "Formatted at 2026-08-12 02:32:02 on dist-test-slave-2w3w"
server_key: "d809b4eb51c7d7418c5befd6c7f867ba"
server_key_iv: "cb14bf5b912433cb44c5468f75b33b6b"
server_key_version: "encryptionkey@0"
I20260812 02:32:02.829676 18340 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786501898713699-14748-0/minicluster-data/ts-1/wal/instance:
uuid: "93a0e7bd61b646869872c5626a0ea41d"
format_stamp: "Formatted at 2026-08-12 02:32:02 on dist-test-slave-2w3w"
server_key: "d809b4eb51c7d7418c5befd6c7f867ba"
server_key_iv: "cb14bf5b912433cb44c5468f75b33b6b"
server_key_version: "encryptionkey@0"
I20260812 02:32:02.834196 18340 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260812 02:32:02.837368 18356 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:32:02.838742 18340 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260812 02:32:02.838898 18340 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786501898713699-14748-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786501898713699-14748-0/minicluster-data/ts-1/wal
uuid: "93a0e7bd61b646869872c5626a0ea41d"
format_stamp: "Formatted at 2026-08-12 02:32:02 on dist-test-slave-2w3w"
server_key: "d809b4eb51c7d7418c5befd6c7f867ba"
server_key_iv: "cb14bf5b912433cb44c5468f75b33b6b"
server_key_version: "encryptionkey@0"
I20260812 02:32:02.839097 18340 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786501898713699-14748-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786501898713699-14748-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786501898713699-14748-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:32:02.867230 18340 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:32:02.871671 18340 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:32:02.872109 18340 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:32:02.873054 18340 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:32:02.874065 18340 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:32:02.874146 18340 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:32:02.874224 18340 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:32:02.874276 18340 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:32:02.885244 18340 rpc_server.cc:307] RPC server started. Bound to: 127.14.103.2:41067
I20260812 02:32:02.885314 18484 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.103.2:41067 every 8 connection(s)
I20260812 02:32:02.886397 18340 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786501898713699-14748-0/minicluster-data/ts-1/data/info.pb
I20260812 02:32:02.896673 14748 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbep4M4/build/debug/bin/kudu as pid 18340
I20260812 02:32:02.896823 14748 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786501898713699-14748-0/minicluster-data/ts-1/wal/instance
Aug 12 02:32:02 dist-test-slave-2w3w krb5kdc[18072](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501922, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.2@KRBTEST.COM for kudu/127.14.103.62@KRBTEST.COM
I20260812 02:32:02.897145 14748 external_mini_cluster.cc:1428] Setting key f2239ec17bedfd6ba671c5fcedd24d90
I20260812 02:32:02.902377 18167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:02.888616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.2:54407 (local address 127.14.103.62:39073)
0812 02:32:02.888778 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:02.888783 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:02.889678 (+ 895us) server_negotiation.cc:407] Connection header received
0812 02:32:02.891211 (+ 1533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:02.891214 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:02.891287 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:02.891398 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:32:02.892666 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:02.893208 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:02.894425 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:02.894652 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:02.894780 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:02.897980 (+ 3200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:32:02.897984 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:32:02.897988 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:32:02.898021 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:32:02.899749 (+ 1728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:02.900499 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:02.900502 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:02.900505 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:02.900606 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:02.900994 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:02.900998 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:02.901000 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:02.901220 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:32:02.901340 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0812 02:32:02.901917 (+ 577us) server_negotiation.cc:299] Negotiation successful
0812 02:32:02.902075 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260812 02:32:02.903633 18487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:02.888987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:39073 (local address 127.14.103.2:54407)
0812 02:32:02.889498 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:02.889546 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0812 02:32:02.890937 (+ 1391us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:32:02.891588 (+ 651us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:32:02.891596 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:32:02.892005 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:32:02.892505 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:02.892519 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:02.893346 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:02.893352 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:02.894280 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:02.894290 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:02.894841 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:02.894846 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:02.895063 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:02.895677 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:32:02.895694 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:32:02.897712 (+ 2018us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:32:02.899998 (+ 2286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:32:02.900005 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:32:02.900026 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:32:02.900370 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:32:02.900708 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:32:02.900711 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:32:02.900713 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:32:02.900888 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:32:02.901348 (+ 460us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:32:02.901357 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:32:02.901651 (+ 294us) client_negotiation.cc:769] Sending connection context
0812 02:32:02.901870 (+ 219us) client_negotiation.cc:240] Negotiation successful
0812 02:32:02.902398 (+ 528us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":113,"threads_started":1}
I20260812 02:32:02.905428 18485 heartbeater.cc:347] Connected to a master server at 127.14.103.62:39073
I20260812 02:32:02.906057 18485 heartbeater.cc:464] Registering TS with master...
I20260812 02:32:02.906659 18485 heartbeater.cc:511] Master 127.14.103.62:39073 requested a full tablet report, sending...
I20260812 02:32:02.908345 18123 ts_manager.cc:194] Registered new tserver with Master: 93a0e7bd61b646869872c5626a0ea41d (127.14.103.2:41067)
WARNING: no policy specified for kudu/127.14.103.3@KRBTEST.COM; defaulting to no policy
I20260812 02:32:02.909113 18123 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.103.2@KRBTEST.COM'} at 127.14.103.2:54407
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.103.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.103.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.3.keytab.
Entry for principal kudu/127.14.103.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.3.keytab.
I20260812 02:32:02.936295 14748 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbep4M4/build/debug/bin/kudu
/tmp/dist-test-taskbep4M4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786501898713699-14748-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786501898713699-14748-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786501898713699-14748-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786501898713699-14748-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.103.3:0
--local_ip_for_outbound_sockets=127.14.103.3
--webserver_interface=127.14.103.3
--webserver_port=0
--tserver_master_addrs=127.14.103.62:39073
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.3.keytab
--principal=kudu/127.14.103.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.103.20:37097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786501898713699-14748-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786501898713699-14748-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786501898713699-14748-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:32:03.067951 18490 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:32:03.068269 18490 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:32:03.068365 18490 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:32:03.073585 18490 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:32:03.073712 18490 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:32:03.073807 18490 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.103.3
I20260812 02:32:03.080207 18490 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.103.20:37097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786501898713699-14748-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786501898713699-14748-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.3.keytab
--principal=kudu/127.14.103.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.103.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786501898713699-14748-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.103.3
--webserver_port=0
--tserver_master_addrs=127.14.103.62:39073
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18490
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.103.3
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786501898713699-14748-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:32:03.081512 18490 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:32:03.082623 18490 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:32:03.094436 18498 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:32:03.094425 18496 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:32:03.094437 18495 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:32:03.095573 18490 server_base.cc:1034] running on GCE node
I20260812 02:32:03.096159 18490 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:32:03.096982 18490 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:32:03.098272 18490 hybrid_clock.cc:648] HybridClock initialized: now 1786501923098209 us; error 78 us; skew 500 ppm
Aug 12 02:32:03 dist-test-slave-2w3w krb5kdc[18072](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501923, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:32:03.102473 18490 init.cc:377] Logged in from keytab as kudu/127.14.103.3@KRBTEST.COM (short username kudu)
I20260812 02:32:03.104202 18490 webserver.cc:460] Webserver started at http://127.14.103.3:41305/ using document root <none> and password file <none>
I20260812 02:32:03.104892 18490 fs_manager.cc:359] Metadata directory not provided
I20260812 02:32:03.104964 18490 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:32:03.105151 18490 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:32:03.107599 18490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786501898713699-14748-0/minicluster-data/ts-2/data/instance:
uuid: "65903f1d17f1473d9a2b44f41aa97c61"
format_stamp: "Formatted at 2026-08-12 02:32:03 on dist-test-slave-2w3w"
server_key: "8153735135aeb9730bbc532ed2179382"
server_key_iv: "1704b416a69df11013d806af3b1201b0"
server_key_version: "encryptionkey@0"
I20260812 02:32:03.108282 18490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786501898713699-14748-0/minicluster-data/ts-2/wal/instance:
uuid: "65903f1d17f1473d9a2b44f41aa97c61"
format_stamp: "Formatted at 2026-08-12 02:32:03 on dist-test-slave-2w3w"
server_key: "8153735135aeb9730bbc532ed2179382"
server_key_iv: "1704b416a69df11013d806af3b1201b0"
server_key_version: "encryptionkey@0"
I20260812 02:32:03.112551 18490 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260812 02:32:03.115561 18506 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:32:03.117328 18490 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 02:32:03.117515 18490 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786501898713699-14748-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786501898713699-14748-0/minicluster-data/ts-2/wal
uuid: "65903f1d17f1473d9a2b44f41aa97c61"
format_stamp: "Formatted at 2026-08-12 02:32:03 on dist-test-slave-2w3w"
server_key: "8153735135aeb9730bbc532ed2179382"
server_key_iv: "1704b416a69df11013d806af3b1201b0"
server_key_version: "encryptionkey@0"
I20260812 02:32:03.117657 18490 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786501898713699-14748-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786501898713699-14748-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786501898713699-14748-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:32:03.133713 18490 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:32:03.138093 18490 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:32:03.138491 18490 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:32:03.139282 18490 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:32:03.140278 18490 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:32:03.140367 18490 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:32:03.140470 18490 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:32:03.140511 18490 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:32:03.153052 18490 rpc_server.cc:307] RPC server started. Bound to: 127.14.103.3:35521
I20260812 02:32:03.153155 18630 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.103.3:35521 every 8 connection(s)
I20260812 02:32:03.154186 18490 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786501898713699-14748-0/minicluster-data/ts-2/data/info.pb
I20260812 02:32:03.155532 14748 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbep4M4/build/debug/bin/kudu as pid 18490
I20260812 02:32:03.155683 14748 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786501898713699-14748-0/minicluster-data/ts-2/wal/instance
I20260812 02:32:03.156008 14748 external_mini_cluster.cc:1428] Setting key ab79597b1f84935921967904f83db9a8
Aug 12 02:32:03 dist-test-slave-2w3w krb5kdc[18072](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501923, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.3@KRBTEST.COM for kudu/127.14.103.62@KRBTEST.COM
I20260812 02:32:03.177665 18167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:03.156831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.3:41785 (local address 127.14.103.62:39073)
0812 02:32:03.156988 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:03.156993 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:03.158423 (+ 1430us) server_negotiation.cc:407] Connection header received
0812 02:32:03.160559 (+ 2136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:03.160562 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:03.160632 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:03.160829 (+ 197us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:32:03.163024 (+ 2195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:03.163592 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:03.165626 (+ 2034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:03.165890 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:03.166026 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:03.170868 (+ 4842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:32:03.170872 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:32:03.170875 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:32:03.170910 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:32:03.173076 (+ 2166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:03.174615 (+ 1539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:03.174621 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:03.174623 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:03.174806 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:03.175584 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:03.175591 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:03.175593 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:03.175936 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:32:03.176085 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0812 02:32:03.177160 (+ 1075us) server_negotiation.cc:299] Negotiation successful
0812 02:32:03.177453 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260812 02:32:03.178644 18633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:03.157393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:39073 (local address 127.14.103.3:41785)
0812 02:32:03.158210 (+ 817us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:03.158282 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0812 02:32:03.160166 (+ 1884us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:32:03.161476 (+ 1310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:32:03.161486 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:32:03.162095 (+ 609us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:32:03.162769 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:03.162786 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:03.163757 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:03.163760 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:03.165435 (+ 1675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:03.165450 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:03.166089 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:03.166093 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:03.166298 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:03.168172 (+ 1874us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:32:03.168203 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:32:03.170582 (+ 2379us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:32:03.173271 (+ 2689us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:32:03.173284 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:32:03.173605 (+ 321us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:32:03.174349 (+ 744us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:32:03.174954 (+ 605us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:32:03.174997 (+ 43us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:32:03.175004 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:32:03.175279 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:32:03.176102 (+ 823us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:32:03.176303 (+ 201us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:32:03.176792 (+ 489us) client_negotiation.cc:769] Sending connection context
0812 02:32:03.177082 (+ 290us) client_negotiation.cc:240] Negotiation successful
0812 02:32:03.177627 (+ 545us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":194,"threads_started":1}
I20260812 02:32:03.180277 18631 heartbeater.cc:347] Connected to a master server at 127.14.103.62:39073
I20260812 02:32:03.180609 18631 heartbeater.cc:464] Registering TS with master...
I20260812 02:32:03.181406 18631 heartbeater.cc:511] Master 127.14.103.62:39073 requested a full tablet report, sending...
I20260812 02:32:03.182816 18123 ts_manager.cc:194] Registered new tserver with Master: 65903f1d17f1473d9a2b44f41aa97c61 (127.14.103.3:35521)
I20260812 02:32:03.183651 18123 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.103.3@KRBTEST.COM'} at 127.14.103.3:41785
I20260812 02:32:03.186601 14748 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 12 02:32:03 dist-test-slave-2w3w krb5kdc[18072](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501923, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Aug 12 02:32:03 dist-test-slave-2w3w krb5kdc[18072](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501923, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.14.103.62@KRBTEST.COM
I20260812 02:32:03.226760 18167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:03.213135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45382 (local address 127.14.103.62:39073)
0812 02:32:03.213300 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:03.213307 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:03.213325 (+ 18us) server_negotiation.cc:407] Connection header received
0812 02:32:03.213384 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:03.213387 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:03.213428 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:03.213529 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:32:03.214309 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:03.214878 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:03.215857 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:03.216090 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:03.216212 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:03.220573 (+ 4361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:32:03.220578 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:32:03.220580 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:32:03.220620 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:32:03.224392 (+ 3772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:03.225131 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:03.225135 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:03.225137 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:03.225251 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:03.225596 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:03.225601 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:03.225607 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:03.225977 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:32:03.226159 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0812 02:32:03.226424 (+ 265us) server_negotiation.cc:299] Negotiation successful
0812 02:32:03.226565 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
W20260812 02:32:03.228008 18123 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:45382
I20260812 02:32:03.230868 14748 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbep4M4/build/debug/bin/kudu with pid 18193
I20260812 02:32:03.240046 14748 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbep4M4/build/debug/bin/kudu with pid 18340
I20260812 02:32:03.248320 14748 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbep4M4/build/debug/bin/kudu with pid 18490
I20260812 02:32:03.256901 14748 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbep4M4/build/debug/bin/kudu with pid 18089
2026-08-12T02:32:03Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2560 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786501898713699-14748-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 02:32:03 dist-test-slave-2w3w krb5kdc[18644](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 02:32:03 dist-test-slave-2w3w krb5kdc[18644](info): set up 2 sockets
Aug 12 02:32:03 dist-test-slave-2w3w krb5kdc[18644](info): commencing operation
krb5kdc: starting...
W20260812 02:32:04.493696 14748 mini_kdc.cc:121] Time spent starting KDC: real 1.209s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786501898713699-14748-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786501898713699-14748-0/krb5kdc/test-user.keytab.
Aug 12 02:32:04 dist-test-slave-2w3w krb5kdc[18644](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501924, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T02:32:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T02:32:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.103.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.103.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.103.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.62.keytab.
Entry for principal kudu/127.14.103.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.62.keytab.
I20260812 02:32:04.676120 14748 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbep4M4/build/debug/bin/kudu
/tmp/dist-test-taskbep4M4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786501898713699-14748-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786501898713699-14748-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786501898713699-14748-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786501898713699-14748-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.103.62:33259
--webserver_interface=127.14.103.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.62.keytab
--principal=kudu/127.14.103.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.103.20:39315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.103.62:33259
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786501898713699-14748-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786501898713699-14748-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786501898713699-14748-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:32:04.815723 18658 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:32:04.816125 18658 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:32:04.816219 18658 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:32:04.821084 18658 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:32:04.821245 18658 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:32:04.821288 18658 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:32:04.821333 18658 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:32:04.821368 18658 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 02:32:04.827632 18658 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.103.20:39315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786501898713699-14748-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786501898713699-14748-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.103.62:33259
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.62.keytab
--principal=kudu/127.14.103.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.103.62:33259
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786501898713699-14748-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.103.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18658
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786501898713699-14748-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:32:04.828938 18658 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:32:04.830065 18658 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:32:04.840886 18667 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:32:04.840972 18665 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:32:04.841281 18658 server_base.cc:1034] running on GCE node
W20260812 02:32:04.841575 18663 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:32:04.842391 18658 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:32:04.843811 18658 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:32:04.845165 18658 hybrid_clock.cc:648] HybridClock initialized: now 1786501924845113 us; error 58 us; skew 500 ppm
Aug 12 02:32:04 dist-test-slave-2w3w krb5kdc[18644](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501924, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:32:04.856508 18658 init.cc:377] Logged in from keytab as kudu/127.14.103.62@KRBTEST.COM (short username kudu)
I20260812 02:32:04.857703 18658 webserver.cc:460] Webserver started at http://127.14.103.62:40323/ using document root <none> and password file <none>
I20260812 02:32:04.858501 18658 fs_manager.cc:359] Metadata directory not provided
I20260812 02:32:04.858629 18658 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:32:04.858919 18658 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:32:04.861341 18658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786501898713699-14748-0/minicluster-data/master-0/data/instance:
uuid: "b13796cae5f24792925a8d90cbc8e043"
format_stamp: "Formatted at 2026-08-12 02:32:04 on dist-test-slave-2w3w"
server_key: "4fa4cf81422ddf8c6a0959e899ce8cee"
server_key_iv: "4ac91155f6f37147b9e580c50e1e229b"
server_key_version: "encryptionkey@0"
I20260812 02:32:04.862027 18658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786501898713699-14748-0/minicluster-data/master-0/wal/instance:
uuid: "b13796cae5f24792925a8d90cbc8e043"
format_stamp: "Formatted at 2026-08-12 02:32:04 on dist-test-slave-2w3w"
server_key: "4fa4cf81422ddf8c6a0959e899ce8cee"
server_key_iv: "4ac91155f6f37147b9e580c50e1e229b"
server_key_version: "encryptionkey@0"
I20260812 02:32:04.867528 18658 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260812 02:32:04.870848 18675 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:32:04.872596 18658 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 02:32:04.872910 18658 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786501898713699-14748-0/minicluster-data/master-0/data,/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786501898713699-14748-0/minicluster-data/master-0/wal
uuid: "b13796cae5f24792925a8d90cbc8e043"
format_stamp: "Formatted at 2026-08-12 02:32:04 on dist-test-slave-2w3w"
server_key: "4fa4cf81422ddf8c6a0959e899ce8cee"
server_key_iv: "4ac91155f6f37147b9e580c50e1e229b"
server_key_version: "encryptionkey@0"
I20260812 02:32:04.873090 18658 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786501898713699-14748-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786501898713699-14748-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786501898713699-14748-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:32:04.901350 18658 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:32:04.905791 18658 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:32:04.906072 18658 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:32:04.915266 18658 rpc_server.cc:307] RPC server started. Bound to: 127.14.103.62:33259
I20260812 02:32:04.915297 18731 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.103.62:33259 every 8 connection(s)
I20260812 02:32:04.916373 18658 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786501898713699-14748-0/minicluster-data/master-0/data/info.pb
I20260812 02:32:04.916698 14748 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbep4M4/build/debug/bin/kudu as pid 18658
I20260812 02:32:04.916836 14748 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786501898713699-14748-0/minicluster-data/master-0/wal/instance
I20260812 02:32:04.917232 14748 external_mini_cluster.cc:1428] Setting key 658ee5ab6807f5a6402373c2b3e4a6c4
I20260812 02:32:04.920080 18732 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 12 02:32:04 dist-test-slave-2w3w krb5kdc[18644](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501924, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.103.62@KRBTEST.COM
I20260812 02:32:04.928629 18732 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b13796cae5f24792925a8d90cbc8e043: Bootstrap starting.
I20260812 02:32:04.931512 18732 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b13796cae5f24792925a8d90cbc8e043: Neither blocks nor log segments found. Creating new log.
I20260812 02:32:04.932544 18732 log.cc:824] T 00000000000000000000000000000000 P b13796cae5f24792925a8d90cbc8e043: Log is configured to *not* fsync() on all Append() calls
I20260812 02:32:04.934526 18735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:04.918714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40264 (local address 127.14.103.62:33259)
0812 02:32:04.919289 (+ 575us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:04.919316 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:04.919367 (+ 51us) server_negotiation.cc:407] Connection header received
0812 02:32:04.920066 (+ 699us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:04.920084 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:04.920586 (+ 502us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:04.920904 (+ 318us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:32:04.921954 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:04.923119 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:04.924366 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:04.925042 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:04.925254 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:04.929130 (+ 3876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:32:04.929136 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:32:04.929152 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:32:04.929190 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:32:04.931192 (+ 2002us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:04.931794 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:04.931800 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:04.931805 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:04.931931 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:04.932276 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:04.932278 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:04.932279 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:04.932903 (+ 624us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:32:04.933125 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0812 02:32:04.933450 (+ 325us) server_negotiation.cc:299] Negotiation successful
0812 02:32:04.933747 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":197,"threads_started":1}
I20260812 02:32:04.935081 18732 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b13796cae5f24792925a8d90cbc8e043: No bootstrap required, opened a new log
I20260812 02:32:04.940488 18732 raft_consensus.cc:348] T 00000000000000000000000000000000 P b13796cae5f24792925a8d90cbc8e043 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b13796cae5f24792925a8d90cbc8e043" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 33259 } }
I20260812 02:32:04.941094 18732 raft_consensus.cc:374] T 00000000000000000000000000000000 P b13796cae5f24792925a8d90cbc8e043 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:32:04.941222 18732 raft_consensus.cc:729] T 00000000000000000000000000000000 P b13796cae5f24792925a8d90cbc8e043 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b13796cae5f24792925a8d90cbc8e043, State: Initialized, Role: FOLLOWER
I20260812 02:32:04.941847 18732 consensus_queue.cc:260] T 00000000000000000000000000000000 P b13796cae5f24792925a8d90cbc8e043 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b13796cae5f24792925a8d90cbc8e043" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 33259 } }
I20260812 02:32:04.942035 18732 raft_consensus.cc:388] T 00000000000000000000000000000000 P b13796cae5f24792925a8d90cbc8e043 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 02:32:04.942137 18732 raft_consensus.cc:482] T 00000000000000000000000000000000 P b13796cae5f24792925a8d90cbc8e043 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 02:32:04.943403 18732 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b13796cae5f24792925a8d90cbc8e043 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:32:04.944734 18732 raft_consensus.cc:504] T 00000000000000000000000000000000 P b13796cae5f24792925a8d90cbc8e043 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b13796cae5f24792925a8d90cbc8e043" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 33259 } }
I20260812 02:32:04.946801 18732 leader_election.cc:302] T 00000000000000000000000000000000 P b13796cae5f24792925a8d90cbc8e043 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b13796cae5f24792925a8d90cbc8e043; no voters:
I20260812 02:32:04.947789 18732 leader_election.cc:288] T 00000000000000000000000000000000 P b13796cae5f24792925a8d90cbc8e043 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 02:32:04.948000 18737 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b13796cae5f24792925a8d90cbc8e043 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:32:04.948343 18737 raft_consensus.cc:686] T 00000000000000000000000000000000 P b13796cae5f24792925a8d90cbc8e043 [term 1 LEADER]: Becoming Leader. State: Replica: b13796cae5f24792925a8d90cbc8e043, State: Running, Role: LEADER
I20260812 02:32:04.948933 18737 consensus_queue.cc:237] T 00000000000000000000000000000000 P b13796cae5f24792925a8d90cbc8e043 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b13796cae5f24792925a8d90cbc8e043" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 33259 } }
I20260812 02:32:04.949575 18732 sys_catalog.cc:564] T 00000000000000000000000000000000 P b13796cae5f24792925a8d90cbc8e043 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:32:04.951160 18740 sys_catalog.cc:455] T 00000000000000000000000000000000 P b13796cae5f24792925a8d90cbc8e043 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b13796cae5f24792925a8d90cbc8e043. Latest consensus state: current_term: 1 leader_uuid: "b13796cae5f24792925a8d90cbc8e043" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b13796cae5f24792925a8d90cbc8e043" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 33259 } } }
I20260812 02:32:04.951332 18740 sys_catalog.cc:458] T 00000000000000000000000000000000 P b13796cae5f24792925a8d90cbc8e043 [sys.catalog]: This master's current role is: LEADER
I20260812 02:32:04.951556 18739 sys_catalog.cc:455] T 00000000000000000000000000000000 P b13796cae5f24792925a8d90cbc8e043 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b13796cae5f24792925a8d90cbc8e043" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b13796cae5f24792925a8d90cbc8e043" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 33259 } } }
I20260812 02:32:04.951711 18739 sys_catalog.cc:458] T 00000000000000000000000000000000 P b13796cae5f24792925a8d90cbc8e043 [sys.catalog]: This master's current role is: LEADER
W20260812 02:32:04.955204 18757 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b13796cae5f24792925a8d90cbc8e043: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 02:32:04.955318 18757 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 02:32:04.955476 18750 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 02:32:04.956404 18750 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 02:32:04.963112 18750 catalog_manager.cc:1347] Generated new cluster ID: d0cc408b45264ca0a8cf20d5bb5e16ff
I20260812 02:32:04.963228 18750 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 02:32:04.975759 18750 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 02:32:04.976826 18750 catalog_manager.cc:1504] Loading token signing keys...
I20260812 02:32:04.987289 18750 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b13796cae5f24792925a8d90cbc8e043: Generated new TSK 0
I20260812 02:32:04.988219 18750 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.103.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.103.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.103.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.1.keytab.
Entry for principal kudu/127.14.103.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.1.keytab.
I20260812 02:32:05.040724 14748 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbep4M4/build/debug/bin/kudu
/tmp/dist-test-taskbep4M4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786501898713699-14748-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786501898713699-14748-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786501898713699-14748-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786501898713699-14748-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.103.1:0
--local_ip_for_outbound_sockets=127.14.103.1
--webserver_interface=127.14.103.1
--webserver_port=0
--tserver_master_addrs=127.14.103.62:33259
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.1.keytab
--principal=kudu/127.14.103.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.103.20:39315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786501898713699-14748-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786501898713699-14748-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786501898713699-14748-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:32:05.191002 18764 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:32:05.191346 18764 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:32:05.191396 18764 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:32:05.195955 18764 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:32:05.196051 18764 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:32:05.196118 18764 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.103.1
I20260812 02:32:05.206231 18764 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.103.20:39315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786501898713699-14748-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786501898713699-14748-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.1.keytab
--principal=kudu/127.14.103.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.103.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786501898713699-14748-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.103.1
--webserver_port=0
--tserver_master_addrs=127.14.103.62:33259
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18764
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.103.1
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786501898713699-14748-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:32:05.207841 18764 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:32:05.209447 18764 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:32:05.225929 18770 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:32:05.226557 18769 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:32:05.226926 18772 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:32:05.227425 18764 server_base.cc:1034] running on GCE node
I20260812 02:32:05.227959 18764 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:32:05.228691 18764 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:32:05.229981 18764 hybrid_clock.cc:648] HybridClock initialized: now 1786501925229846 us; error 135 us; skew 500 ppm
Aug 12 02:32:05 dist-test-slave-2w3w krb5kdc[18644](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501925, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:32:05.234016 18764 init.cc:377] Logged in from keytab as kudu/127.14.103.1@KRBTEST.COM (short username kudu)
I20260812 02:32:05.235252 18764 webserver.cc:460] Webserver started at http://127.14.103.1:44781/ using document root <none> and password file <none>
I20260812 02:32:05.235898 18764 fs_manager.cc:359] Metadata directory not provided
I20260812 02:32:05.235968 18764 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:32:05.236202 18764 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:32:05.238535 18764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786501898713699-14748-0/minicluster-data/ts-0/data/instance:
uuid: "20b664cf9d1747628727f1ce06242abe"
format_stamp: "Formatted at 2026-08-12 02:32:05 on dist-test-slave-2w3w"
server_key: "cf19de3050d3453c37e4d6dd40905ae1"
server_key_iv: "2086d3a21d5719138798a6eccebe1daa"
server_key_version: "encryptionkey@0"
I20260812 02:32:05.239241 18764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786501898713699-14748-0/minicluster-data/ts-0/wal/instance:
uuid: "20b664cf9d1747628727f1ce06242abe"
format_stamp: "Formatted at 2026-08-12 02:32:05 on dist-test-slave-2w3w"
server_key: "cf19de3050d3453c37e4d6dd40905ae1"
server_key_iv: "2086d3a21d5719138798a6eccebe1daa"
server_key_version: "encryptionkey@0"
I20260812 02:32:05.245105 18764 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260812 02:32:05.248679 18780 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:32:05.250296 18764 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 02:32:05.250511 18764 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786501898713699-14748-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786501898713699-14748-0/minicluster-data/ts-0/wal
uuid: "20b664cf9d1747628727f1ce06242abe"
format_stamp: "Formatted at 2026-08-12 02:32:05 on dist-test-slave-2w3w"
server_key: "cf19de3050d3453c37e4d6dd40905ae1"
server_key_iv: "2086d3a21d5719138798a6eccebe1daa"
server_key_version: "encryptionkey@0"
I20260812 02:32:05.250658 18764 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786501898713699-14748-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786501898713699-14748-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786501898713699-14748-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:32:05.265544 18764 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:32:05.269452 18764 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:32:05.269750 18764 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:32:05.270485 18764 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:32:05.271613 18764 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:32:05.271705 18764 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:32:05.271786 18764 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:32:05.271847 18764 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:32:05.283063 18764 rpc_server.cc:307] RPC server started. Bound to: 127.14.103.1:36249
I20260812 02:32:05.283079 18903 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.103.1:36249 every 8 connection(s)
I20260812 02:32:05.284718 18764 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786501898713699-14748-0/minicluster-data/ts-0/data/info.pb
I20260812 02:32:05.293247 14748 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbep4M4/build/debug/bin/kudu as pid 18764
I20260812 02:32:05.293396 14748 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786501898713699-14748-0/minicluster-data/ts-0/wal/instance
I20260812 02:32:05.293735 14748 external_mini_cluster.cc:1428] Setting key e533f41a7af96f161dcefcf76aba70cb
Aug 12 02:32:05 dist-test-slave-2w3w krb5kdc[18644](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501925, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.1@KRBTEST.COM for kudu/127.14.103.62@KRBTEST.COM
I20260812 02:32:05.305752 18735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:05.287488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.1:49965 (local address 127.14.103.62:33259)
0812 02:32:05.287652 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:05.287656 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:05.288554 (+ 898us) server_negotiation.cc:407] Connection header received
0812 02:32:05.290177 (+ 1623us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:05.290181 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:05.290251 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:05.290375 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:32:05.291855 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:05.292477 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:05.293952 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:05.294302 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:05.294442 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:05.298207 (+ 3765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:32:05.298210 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:32:05.298213 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:32:05.298241 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:32:05.300707 (+ 2466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:05.301667 (+ 960us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:05.301673 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:05.301680 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:05.301794 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:05.302252 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:05.302257 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:05.302259 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:05.302499 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:32:05.302622 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0812 02:32:05.303973 (+ 1351us) server_negotiation.cc:299] Negotiation successful
0812 02:32:05.304809 (+ 836us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260812 02:32:05.307540 18907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:05.287801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:33259 (local address 127.14.103.1:49965)
0812 02:32:05.288361 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:05.288424 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0812 02:32:05.289876 (+ 1452us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:32:05.290580 (+ 704us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:32:05.290588 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:32:05.291103 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:32:05.291686 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:05.291702 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:05.292643 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:05.292647 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:05.293752 (+ 1105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:05.293765 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:05.294519 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:05.294524 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:05.294737 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:05.295644 (+ 907us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:32:05.295667 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:32:05.297953 (+ 2286us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:32:05.300946 (+ 2993us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:32:05.300952 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:32:05.300972 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:32:05.301489 (+ 517us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:32:05.301906 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:32:05.301909 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:32:05.301911 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:32:05.302131 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:32:05.302623 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:32:05.302631 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:32:05.302952 (+ 321us) client_negotiation.cc:769] Sending connection context
0812 02:32:05.303792 (+ 840us) client_negotiation.cc:240] Negotiation successful
0812 02:32:05.305800 (+ 2008us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":137,"threads_started":1}
WARNING: no policy specified for kudu/127.14.103.2@KRBTEST.COM; defaulting to no policy
I20260812 02:32:05.309212 18905 heartbeater.cc:347] Connected to a master server at 127.14.103.62:33259
I20260812 02:32:05.309504 18905 heartbeater.cc:464] Registering TS with master...
I20260812 02:32:05.310217 18905 heartbeater.cc:511] Master 127.14.103.62:33259 requested a full tablet report, sending...
I20260812 02:32:05.312332 18694 ts_manager.cc:194] Registered new tserver with Master: 20b664cf9d1747628727f1ce06242abe (127.14.103.1:36249)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.103.2@KRBTEST.COM" created.
I20260812 02:32:05.313722 18694 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.103.1@KRBTEST.COM'} at 127.14.103.1:49965
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.103.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.2.keytab.
Entry for principal kudu/127.14.103.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.2.keytab.
I20260812 02:32:05.336344 14748 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbep4M4/build/debug/bin/kudu
/tmp/dist-test-taskbep4M4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786501898713699-14748-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786501898713699-14748-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786501898713699-14748-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786501898713699-14748-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.103.2:0
--local_ip_for_outbound_sockets=127.14.103.2
--webserver_interface=127.14.103.2
--webserver_port=0
--tserver_master_addrs=127.14.103.62:33259
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.2.keytab
--principal=kudu/127.14.103.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.103.20:39315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786501898713699-14748-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786501898713699-14748-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786501898713699-14748-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:32:05.466755 18910 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:32:05.467221 18910 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:32:05.467284 18910 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:32:05.471953 18910 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:32:05.472038 18910 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:32:05.472105 18910 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.103.2
I20260812 02:32:05.479053 18910 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.103.20:39315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786501898713699-14748-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786501898713699-14748-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.2.keytab
--principal=kudu/127.14.103.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.103.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786501898713699-14748-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.103.2
--webserver_port=0
--tserver_master_addrs=127.14.103.62:33259
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18910
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.103.2
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786501898713699-14748-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:32:05.480337 18910 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:32:05.481932 18910 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:32:05.494482 18917 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:32:05.494433 18919 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:32:05.494463 18916 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:32:05.495215 18910 server_base.cc:1034] running on GCE node
I20260812 02:32:05.495879 18910 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:32:05.496628 18910 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:32:05.497812 18910 hybrid_clock.cc:648] HybridClock initialized: now 1786501925497766 us; error 65 us; skew 500 ppm
Aug 12 02:32:05 dist-test-slave-2w3w krb5kdc[18644](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501925, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:32:05.504376 18910 init.cc:377] Logged in from keytab as kudu/127.14.103.2@KRBTEST.COM (short username kudu)
I20260812 02:32:05.506052 18910 webserver.cc:460] Webserver started at http://127.14.103.2:44945/ using document root <none> and password file <none>
I20260812 02:32:05.506933 18910 fs_manager.cc:359] Metadata directory not provided
I20260812 02:32:05.507095 18910 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:32:05.507355 18910 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:32:05.509732 18910 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786501898713699-14748-0/minicluster-data/ts-1/data/instance:
uuid: "0293b7ac216d4040bde5b03808952a07"
format_stamp: "Formatted at 2026-08-12 02:32:05 on dist-test-slave-2w3w"
server_key: "59e8fd08fef5ce091e115f953d0ef41d"
server_key_iv: "34df78cf87c0661f4600e6daab2286ab"
server_key_version: "encryptionkey@0"
I20260812 02:32:05.510428 18910 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786501898713699-14748-0/minicluster-data/ts-1/wal/instance:
uuid: "0293b7ac216d4040bde5b03808952a07"
format_stamp: "Formatted at 2026-08-12 02:32:05 on dist-test-slave-2w3w"
server_key: "59e8fd08fef5ce091e115f953d0ef41d"
server_key_iv: "34df78cf87c0661f4600e6daab2286ab"
server_key_version: "encryptionkey@0"
I20260812 02:32:05.515178 18910 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.008s sys 0.000s
I20260812 02:32:05.518227 18927 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:32:05.519904 18910 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 02:32:05.520038 18910 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786501898713699-14748-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786501898713699-14748-0/minicluster-data/ts-1/wal
uuid: "0293b7ac216d4040bde5b03808952a07"
format_stamp: "Formatted at 2026-08-12 02:32:05 on dist-test-slave-2w3w"
server_key: "59e8fd08fef5ce091e115f953d0ef41d"
server_key_iv: "34df78cf87c0661f4600e6daab2286ab"
server_key_version: "encryptionkey@0"
I20260812 02:32:05.520208 18910 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786501898713699-14748-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786501898713699-14748-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786501898713699-14748-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:32:05.548826 18910 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:32:05.553584 18910 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:32:05.553893 18910 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:32:05.554668 18910 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:32:05.555691 18910 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:32:05.555771 18910 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:32:05.555836 18910 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:32:05.555892 18910 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:32:05.568537 18910 rpc_server.cc:307] RPC server started. Bound to: 127.14.103.2:37161
I20260812 02:32:05.568555 19055 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.103.2:37161 every 8 connection(s)
I20260812 02:32:05.569615 18910 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786501898713699-14748-0/minicluster-data/ts-1/data/info.pb
I20260812 02:32:05.576694 14748 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbep4M4/build/debug/bin/kudu as pid 18910
I20260812 02:32:05.576977 14748 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786501898713699-14748-0/minicluster-data/ts-1/wal/instance
I20260812 02:32:05.577307 14748 external_mini_cluster.cc:1428] Setting key 73c2d722d4dfe423343b75bf1724de37
Aug 12 02:32:05 dist-test-slave-2w3w krb5kdc[18644](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501925, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.2@KRBTEST.COM for kudu/127.14.103.62@KRBTEST.COM
I20260812 02:32:05.590581 18735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:05.571785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.2:44403 (local address 127.14.103.62:33259)
0812 02:32:05.571996 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:05.571999 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:05.572824 (+ 825us) server_negotiation.cc:407] Connection header received
0812 02:32:05.574361 (+ 1537us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:05.574365 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:05.574430 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:05.574506 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:32:05.575980 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:05.576634 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:05.578688 (+ 2054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:05.579079 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:05.579354 (+ 275us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:05.584018 (+ 4664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:32:05.584022 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:32:05.584024 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:32:05.584056 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:32:05.587396 (+ 3340us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:05.588462 (+ 1066us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:05.588466 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:05.588467 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:05.588568 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:05.589069 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:05.589071 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:05.589072 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:05.589321 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:32:05.589404 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0812 02:32:05.590293 (+ 889us) server_negotiation.cc:299] Negotiation successful
0812 02:32:05.590442 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112}
WARNING: no policy specified for kudu/127.14.103.3@KRBTEST.COM; defaulting to no policy
I20260812 02:32:05.592139 19059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:05.572135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:33259 (local address 127.14.103.2:44403)
0812 02:32:05.572649 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:05.572698 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0812 02:32:05.574113 (+ 1415us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:32:05.574694 (+ 581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:32:05.574702 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:32:05.575235 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:32:05.575791 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:05.575807 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:05.577001 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:05.577005 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:05.578515 (+ 1510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:05.578530 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:05.580329 (+ 1799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:05.580335 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:05.580541 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:05.581376 (+ 835us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:32:05.581395 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:32:05.583711 (+ 2316us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:32:05.587660 (+ 3949us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:32:05.587673 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:32:05.587699 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:32:05.588270 (+ 571us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:32:05.588695 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:32:05.588701 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:32:05.588705 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:32:05.588945 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:32:05.589444 (+ 499us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:32:05.589455 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:32:05.589891 (+ 436us) client_negotiation.cc:769] Sending connection context
0812 02:32:05.590179 (+ 288us) client_negotiation.cc:240] Negotiation successful
0812 02:32:05.590621 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":136,"threads_started":1}
I20260812 02:32:05.593784 19056 heartbeater.cc:347] Connected to a master server at 127.14.103.62:33259
I20260812 02:32:05.594050 19056 heartbeater.cc:464] Registering TS with master...
I20260812 02:32:05.594606 19056 heartbeater.cc:511] Master 127.14.103.62:33259 requested a full tablet report, sending...
I20260812 02:32:05.595896 18694 ts_manager.cc:194] Registered new tserver with Master: 0293b7ac216d4040bde5b03808952a07 (127.14.103.2:37161)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.103.3@KRBTEST.COM" created.
I20260812 02:32:05.596877 18694 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.103.2@KRBTEST.COM'} at 127.14.103.2:44403
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.103.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.3.keytab.
Entry for principal kudu/127.14.103.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.3.keytab.
I20260812 02:32:05.621532 14748 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbep4M4/build/debug/bin/kudu
/tmp/dist-test-taskbep4M4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786501898713699-14748-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786501898713699-14748-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786501898713699-14748-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786501898713699-14748-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.103.3:0
--local_ip_for_outbound_sockets=127.14.103.3
--webserver_interface=127.14.103.3
--webserver_port=0
--tserver_master_addrs=127.14.103.62:33259
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.3.keytab
--principal=kudu/127.14.103.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.103.20:39315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786501898713699-14748-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786501898713699-14748-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786501898713699-14748-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:32:05.753783 19062 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:32:05.754124 19062 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:32:05.754172 19062 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:32:05.758924 19062 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:32:05.759132 19062 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:32:05.759239 19062 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.103.3
I20260812 02:32:05.765672 19062 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.103.20:39315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786501898713699-14748-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786501898713699-14748-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.3.keytab
--principal=kudu/127.14.103.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.103.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786501898713699-14748-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.103.3
--webserver_port=0
--tserver_master_addrs=127.14.103.62:33259
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19062
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.103.3
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786501898713699-14748-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:32:05.767105 19062 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:32:05.768324 19062 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:32:05.779968 19071 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:32:05.779954 19067 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:32:05.780097 19068 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:32:05.780732 19062 server_base.cc:1034] running on GCE node
I20260812 02:32:05.781248 19062 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:32:05.782078 19062 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:32:05.785455 19062 hybrid_clock.cc:648] HybridClock initialized: now 1786501925785416 us; error 35 us; skew 500 ppm
Aug 12 02:32:05 dist-test-slave-2w3w krb5kdc[18644](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501925, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:32:05.790208 19062 init.cc:377] Logged in from keytab as kudu/127.14.103.3@KRBTEST.COM (short username kudu)
I20260812 02:32:05.792060 19062 webserver.cc:460] Webserver started at http://127.14.103.3:41855/ using document root <none> and password file <none>
I20260812 02:32:05.793109 19062 fs_manager.cc:359] Metadata directory not provided
I20260812 02:32:05.793294 19062 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:32:05.793605 19062 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:32:05.796442 19062 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786501898713699-14748-0/minicluster-data/ts-2/data/instance:
uuid: "9e015f5069e54ab8a9f8560da8df82ff"
format_stamp: "Formatted at 2026-08-12 02:32:05 on dist-test-slave-2w3w"
server_key: "98c6babce39c2ac075d01ce87c74c4e4"
server_key_iv: "a305b0ba03d2aeb64643a1fe9f7ddb13"
server_key_version: "encryptionkey@0"
I20260812 02:32:05.797382 19062 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786501898713699-14748-0/minicluster-data/ts-2/wal/instance:
uuid: "9e015f5069e54ab8a9f8560da8df82ff"
format_stamp: "Formatted at 2026-08-12 02:32:05 on dist-test-slave-2w3w"
server_key: "98c6babce39c2ac075d01ce87c74c4e4"
server_key_iv: "a305b0ba03d2aeb64643a1fe9f7ddb13"
server_key_version: "encryptionkey@0"
I20260812 02:32:05.803269 19062 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260812 02:32:05.806732 19079 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:32:05.808665 19062 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 02:32:05.808876 19062 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786501898713699-14748-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786501898713699-14748-0/minicluster-data/ts-2/wal
uuid: "9e015f5069e54ab8a9f8560da8df82ff"
format_stamp: "Formatted at 2026-08-12 02:32:05 on dist-test-slave-2w3w"
server_key: "98c6babce39c2ac075d01ce87c74c4e4"
server_key_iv: "a305b0ba03d2aeb64643a1fe9f7ddb13"
server_key_version: "encryptionkey@0"
I20260812 02:32:05.809043 19062 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786501898713699-14748-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786501898713699-14748-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786501898713699-14748-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:32:05.832952 19062 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:32:05.836725 19062 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:32:05.837033 19062 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:32:05.837740 19062 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:32:05.838701 19062 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:32:05.838752 19062 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:32:05.838795 19062 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:32:05.838811 19062 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:32:05.851573 19062 rpc_server.cc:307] RPC server started. Bound to: 127.14.103.3:34155
I20260812 02:32:05.851692 19203 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.103.3:34155 every 8 connection(s)
I20260812 02:32:05.853036 19062 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786501898713699-14748-0/minicluster-data/ts-2/data/info.pb
I20260812 02:32:05.863425 14748 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbep4M4/build/debug/bin/kudu as pid 19062
I20260812 02:32:05.863588 14748 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786501898713699-14748-0/minicluster-data/ts-2/wal/instance
I20260812 02:32:05.863976 14748 external_mini_cluster.cc:1428] Setting key b2ec9096c9b600ea5ffa36c2565eeece
Aug 12 02:32:05 dist-test-slave-2w3w krb5kdc[18644](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501925, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.3@KRBTEST.COM for kudu/127.14.103.62@KRBTEST.COM
I20260812 02:32:05.872500 18735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:05.856328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.3:53503 (local address 127.14.103.62:33259)
0812 02:32:05.856678 (+ 350us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:05.856681 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:05.857541 (+ 860us) server_negotiation.cc:407] Connection header received
0812 02:32:05.859146 (+ 1605us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:05.859151 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:05.859222 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:05.859320 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:32:05.860723 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:05.861356 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:05.862730 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:05.863079 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:05.863186 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:05.867674 (+ 4488us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:32:05.867678 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:32:05.867680 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:32:05.867711 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:32:05.869539 (+ 1828us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:05.870235 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:05.870238 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:05.870239 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:05.870330 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:05.870759 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:05.870761 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:05.870762 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:05.871147 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:32:05.871242 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0812 02:32:05.872208 (+ 966us) server_negotiation.cc:299] Negotiation successful
0812 02:32:05.872365 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":235}
I20260812 02:32:05.873557 19207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:05.856761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:33259 (local address 127.14.103.3:53503)
0812 02:32:05.857356 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:05.857407 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0812 02:32:05.858783 (+ 1376us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:32:05.859557 (+ 774us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:32:05.859566 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:32:05.859993 (+ 427us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:32:05.860527 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:05.860542 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:05.861522 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:05.861526 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:05.862572 (+ 1046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:05.862584 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:05.863320 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:05.863327 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:05.863587 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:05.864964 (+ 1377us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:32:05.864992 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:32:05.867475 (+ 2483us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:32:05.869719 (+ 2244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:32:05.869726 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:32:05.869743 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:32:05.870111 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:32:05.870467 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:32:05.870470 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:32:05.870472 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:32:05.870646 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:32:05.871302 (+ 656us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:32:05.871309 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:32:05.871634 (+ 325us) client_negotiation.cc:769] Sending connection context
0812 02:32:05.871855 (+ 221us) client_negotiation.cc:240] Negotiation successful
0812 02:32:05.872229 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":160,"threads_started":1}
I20260812 02:32:05.875902 19204 heartbeater.cc:347] Connected to a master server at 127.14.103.62:33259
I20260812 02:32:05.876370 19204 heartbeater.cc:464] Registering TS with master...
I20260812 02:32:05.877086 19204 heartbeater.cc:511] Master 127.14.103.62:33259 requested a full tablet report, sending...
I20260812 02:32:05.878822 18694 ts_manager.cc:194] Registered new tserver with Master: 9e015f5069e54ab8a9f8560da8df82ff (127.14.103.3:34155)
I20260812 02:32:05.879563 18694 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.103.3@KRBTEST.COM'} at 127.14.103.3:53503
I20260812 02:32:05.880833 14748 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 12 02:32:05 dist-test-slave-2w3w krb5kdc[18644](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501925, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 12 02:32:05 dist-test-slave-2w3w krb5kdc[18644](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501925, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.103.1@KRBTEST.COM
I20260812 02:32:05.921327 19213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:05.907389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45678 (local address 127.14.103.1:36249)
0812 02:32:05.907813 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:05.907824 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:05.907856 (+ 32us) server_negotiation.cc:407] Connection header received
0812 02:32:05.908068 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:05.908075 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:05.908307 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:05.908484 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:32:05.909404 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:05.910427 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:05.911598 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:05.912083 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:05.912237 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:05.915667 (+ 3430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:32:05.915691 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:32:05.915706 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:32:05.915748 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:32:05.917889 (+ 2141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:05.919262 (+ 1373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:05.919271 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:05.919273 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:05.919412 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:05.919971 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:05.919974 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:05.919976 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:05.920365 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:32:05.920608 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0812 02:32:05.920886 (+ 278us) server_negotiation.cc:299] Negotiation successful
0812 02:32:05.921112 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":149,"threads_started":1}
Aug 12 02:32:05 dist-test-slave-2w3w krb5kdc[18644](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501925, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.103.62@KRBTEST.COM
I20260812 02:32:05.944921 18735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:05.927118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40272 (local address 127.14.103.62:33259)
0812 02:32:05.927286 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:05.927290 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:05.927434 (+ 144us) server_negotiation.cc:407] Connection header received
0812 02:32:05.927492 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:05.927495 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:05.927543 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:05.927624 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:32:05.929070 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:05.930217 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:05.931722 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:05.932176 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:05.932316 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:05.936126 (+ 3810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:32:05.936131 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:32:05.936133 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:32:05.936171 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:32:05.938784 (+ 2613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:05.940006 (+ 1222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:05.940011 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:05.940012 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:05.940358 (+ 346us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:05.940985 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:05.940992 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:05.940994 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:05.941527 (+ 533us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:32:05.942027 (+ 500us) server_negotiation.cc:1035] Waiting for connection context
0812 02:32:05.942397 (+ 370us) server_negotiation.cc:299] Negotiation successful
0812 02:32:05.943909 (+ 1512us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
W20260812 02:32:05.946154 18694 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40272
I20260812 02:32:05.947872 14748 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbep4M4/build/debug/bin/kudu with pid 18764
I20260812 02:32:05.959097 14748 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbep4M4/build/debug/bin/kudu with pid 18910
I20260812 02:32:05.978065 14748 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbep4M4/build/debug/bin/kudu with pid 19062
I20260812 02:32:05.988436 14748 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbep4M4/build/debug/bin/kudu with pid 18658
2026-08-12T02:32:05Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2730 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501898713699-14748-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 02:32:06 dist-test-slave-2w3w krb5kdc[19219](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 02:32:06 dist-test-slave-2w3w krb5kdc[19219](info): set up 2 sockets
Aug 12 02:32:06 dist-test-slave-2w3w krb5kdc[19219](info): commencing operation
krb5kdc: starting...
W20260812 02:32:07.218636 14748 mini_kdc.cc:121] Time spent starting KDC: real 1.203s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501898713699-14748-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501898713699-14748-0/krb5kdc/test-user.keytab.
Aug 12 02:32:07 dist-test-slave-2w3w krb5kdc[19219](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501927, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T02:32:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T02:32:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.103.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.103.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.103.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.62.keytab.
Entry for principal kudu/127.14.103.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.62.keytab.
I20260812 02:32:07.405285 14748 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbep4M4/build/debug/bin/kudu
/tmp/dist-test-taskbep4M4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501898713699-14748-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501898713699-14748-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501898713699-14748-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501898713699-14748-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.103.62:32917
--webserver_interface=127.14.103.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.62.keytab
--principal=kudu/127.14.103.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.103.20:42651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.103.62:32917
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501898713699-14748-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501898713699-14748-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501898713699-14748-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:32:07.541000 19234 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:32:07.541344 19234 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:32:07.541393 19234 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:32:07.546855 19234 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:32:07.547027 19234 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:32:07.547055 19234 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:32:07.547075 19234 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:32:07.547248 19234 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 02:32:07.554131 19234 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.103.20:42651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501898713699-14748-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501898713699-14748-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.103.62:32917
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.62.keytab
--principal=kudu/127.14.103.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.103.62:32917
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501898713699-14748-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.103.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19234
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501898713699-14748-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:32:07.555462 19234 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:32:07.556588 19234 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:32:07.566371 19240 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:32:07.566402 19242 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:32:07.566865 19234 server_base.cc:1034] running on GCE node
W20260812 02:32:07.566371 19244 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:32:07.567799 19234 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:32:07.569061 19234 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:32:07.570286 19234 hybrid_clock.cc:648] HybridClock initialized: now 1786501927570272 us; error 33 us; skew 500 ppm
Aug 12 02:32:07 dist-test-slave-2w3w krb5kdc[19219](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501927, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:32:07.573779 19234 init.cc:377] Logged in from keytab as kudu/127.14.103.62@KRBTEST.COM (short username kudu)
I20260812 02:32:07.574676 19234 fs_manager.cc:359] Metadata directory not provided
I20260812 02:32:07.574826 19234 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:32:07.575141 19234 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:32:07.577476 19234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501898713699-14748-0/minicluster-data/master-0/data/instance:
uuid: "f696c7d60e1f48c1bbff12b98714e5cc"
format_stamp: "Formatted at 2026-08-12 02:32:07 on dist-test-slave-2w3w"
server_key: "6533c85b1be5685ccfd6f215f2f524fb"
server_key_iv: "0d72946efb85ba789bb078e43da5e8b9"
server_key_version: "encryptionkey@0"
I20260812 02:32:07.578130 19234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501898713699-14748-0/minicluster-data/master-0/wal/instance:
uuid: "f696c7d60e1f48c1bbff12b98714e5cc"
format_stamp: "Formatted at 2026-08-12 02:32:07 on dist-test-slave-2w3w"
server_key: "6533c85b1be5685ccfd6f215f2f524fb"
server_key_iv: "0d72946efb85ba789bb078e43da5e8b9"
server_key_version: "encryptionkey@0"
I20260812 02:32:07.582440 19234 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260812 02:32:07.585925 19250 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:32:07.587764 19234 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 02:32:07.587967 19234 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501898713699-14748-0/minicluster-data/master-0/data,/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501898713699-14748-0/minicluster-data/master-0/wal
uuid: "f696c7d60e1f48c1bbff12b98714e5cc"
format_stamp: "Formatted at 2026-08-12 02:32:07 on dist-test-slave-2w3w"
server_key: "6533c85b1be5685ccfd6f215f2f524fb"
server_key_iv: "0d72946efb85ba789bb078e43da5e8b9"
server_key_version: "encryptionkey@0"
I20260812 02:32:07.588150 19234 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501898713699-14748-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501898713699-14748-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501898713699-14748-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:32:07.620018 19234 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:32:07.624874 19234 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:32:07.625133 19234 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:32:07.636593 19234 rpc_server.cc:307] RPC server started. Bound to: 127.14.103.62:32917
I20260812 02:32:07.636850 19305 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.103.62:32917 every 8 connection(s)
I20260812 02:32:07.637557 19234 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501898713699-14748-0/minicluster-data/master-0/data/info.pb
I20260812 02:32:07.641476 19306 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:32:07.645779 14748 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbep4M4/build/debug/bin/kudu as pid 19234
I20260812 02:32:07.646013 14748 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501898713699-14748-0/minicluster-data/master-0/wal/instance
I20260812 02:32:07.646567 14748 external_mini_cluster.cc:1428] Setting key 4f19e27131cf4276e5fcd83fd8df0ed1
I20260812 02:32:07.649214 19306 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f696c7d60e1f48c1bbff12b98714e5cc: Bootstrap starting.
I20260812 02:32:07.652048 19306 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f696c7d60e1f48c1bbff12b98714e5cc: Neither blocks nor log segments found. Creating new log.
I20260812 02:32:07.653455 19306 log.cc:824] T 00000000000000000000000000000000 P f696c7d60e1f48c1bbff12b98714e5cc: Log is configured to *not* fsync() on all Append() calls
I20260812 02:32:07.656662 19306 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f696c7d60e1f48c1bbff12b98714e5cc: No bootstrap required, opened a new log
Aug 12 02:32:07 dist-test-slave-2w3w krb5kdc[19219](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501927, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.103.62@KRBTEST.COM
I20260812 02:32:07.662407 19306 raft_consensus.cc:348] T 00000000000000000000000000000000 P f696c7d60e1f48c1bbff12b98714e5cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f696c7d60e1f48c1bbff12b98714e5cc" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 32917 } }
I20260812 02:32:07.662729 19306 raft_consensus.cc:374] T 00000000000000000000000000000000 P f696c7d60e1f48c1bbff12b98714e5cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:32:07.663129 19306 raft_consensus.cc:729] T 00000000000000000000000000000000 P f696c7d60e1f48c1bbff12b98714e5cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f696c7d60e1f48c1bbff12b98714e5cc, State: Initialized, Role: FOLLOWER
I20260812 02:32:07.663813 19306 consensus_queue.cc:260] T 00000000000000000000000000000000 P f696c7d60e1f48c1bbff12b98714e5cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f696c7d60e1f48c1bbff12b98714e5cc" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 32917 } }
I20260812 02:32:07.663990 19306 raft_consensus.cc:388] T 00000000000000000000000000000000 P f696c7d60e1f48c1bbff12b98714e5cc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 02:32:07.664088 19306 raft_consensus.cc:482] T 00000000000000000000000000000000 P f696c7d60e1f48c1bbff12b98714e5cc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 02:32:07.664209 19306 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f696c7d60e1f48c1bbff12b98714e5cc [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:32:07.665688 19306 raft_consensus.cc:504] T 00000000000000000000000000000000 P f696c7d60e1f48c1bbff12b98714e5cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f696c7d60e1f48c1bbff12b98714e5cc" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 32917 } }
I20260812 02:32:07.666181 19306 leader_election.cc:302] T 00000000000000000000000000000000 P f696c7d60e1f48c1bbff12b98714e5cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f696c7d60e1f48c1bbff12b98714e5cc; no voters:
I20260812 02:32:07.666636 19306 leader_election.cc:288] T 00000000000000000000000000000000 P f696c7d60e1f48c1bbff12b98714e5cc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 02:32:07.666839 19311 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f696c7d60e1f48c1bbff12b98714e5cc [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:32:07.667244 19311 raft_consensus.cc:686] T 00000000000000000000000000000000 P f696c7d60e1f48c1bbff12b98714e5cc [term 1 LEADER]: Becoming Leader. State: Replica: f696c7d60e1f48c1bbff12b98714e5cc, State: Running, Role: LEADER
I20260812 02:32:07.667735 19311 consensus_queue.cc:237] T 00000000000000000000000000000000 P f696c7d60e1f48c1bbff12b98714e5cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f696c7d60e1f48c1bbff12b98714e5cc" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 32917 } }
I20260812 02:32:07.668161 19306 sys_catalog.cc:564] T 00000000000000000000000000000000 P f696c7d60e1f48c1bbff12b98714e5cc [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:32:07.668318 19309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:07.648313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39340 (local address 127.14.103.62:32917)
0812 02:32:07.649332 (+ 1019us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:07.649363 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:07.649700 (+ 337us) server_negotiation.cc:407] Connection header received
0812 02:32:07.651077 (+ 1377us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:07.651215 (+ 138us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:07.651819 (+ 604us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:07.652458 (+ 639us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:32:07.653814 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:07.655367 (+ 1553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:07.657059 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:07.657913 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:07.658239 (+ 326us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:07.661751 (+ 3512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:32:07.661758 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:32:07.661778 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:32:07.661818 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:32:07.663948 (+ 2130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:07.664649 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:07.664677 (+ 28us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:07.664684 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:07.664870 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:07.665304 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:07.665308 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:07.665310 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:07.665969 (+ 659us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:32:07.666180 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0812 02:32:07.666840 (+ 660us) server_negotiation.cc:299] Negotiation successful
0812 02:32:07.667268 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":632,"thread_start_us":420,"threads_started":1}
I20260812 02:32:07.669204 19312 sys_catalog.cc:455] T 00000000000000000000000000000000 P f696c7d60e1f48c1bbff12b98714e5cc [sys.catalog]: SysCatalogTable state changed. Reason: New leader f696c7d60e1f48c1bbff12b98714e5cc. Latest consensus state: current_term: 1 leader_uuid: "f696c7d60e1f48c1bbff12b98714e5cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f696c7d60e1f48c1bbff12b98714e5cc" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 32917 } } }
I20260812 02:32:07.669349 19312 sys_catalog.cc:458] T 00000000000000000000000000000000 P f696c7d60e1f48c1bbff12b98714e5cc [sys.catalog]: This master's current role is: LEADER
I20260812 02:32:07.670353 19314 sys_catalog.cc:455] T 00000000000000000000000000000000 P f696c7d60e1f48c1bbff12b98714e5cc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f696c7d60e1f48c1bbff12b98714e5cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f696c7d60e1f48c1bbff12b98714e5cc" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 32917 } } }
I20260812 02:32:07.671639 19321 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 02:32:07.672019 19314 sys_catalog.cc:458] T 00000000000000000000000000000000 P f696c7d60e1f48c1bbff12b98714e5cc [sys.catalog]: This master's current role is: LEADER
I20260812 02:32:07.675448 19321 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 02:32:07.681828 19321 catalog_manager.cc:1347] Generated new cluster ID: 893a5854ff9e4fb99a7bd2738e9e8259
I20260812 02:32:07.681972 19321 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 02:32:07.698246 19321 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 02:32:07.700366 19321 catalog_manager.cc:1504] Loading token signing keys...
I20260812 02:32:07.709805 19321 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f696c7d60e1f48c1bbff12b98714e5cc: Generated new TSK 0
I20260812 02:32:07.710765 19321 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.103.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.103.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.103.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.1.keytab.
Entry for principal kudu/127.14.103.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.1.keytab.
I20260812 02:32:07.764439 14748 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbep4M4/build/debug/bin/kudu
/tmp/dist-test-taskbep4M4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501898713699-14748-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501898713699-14748-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501898713699-14748-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501898713699-14748-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.103.1:0
--local_ip_for_outbound_sockets=127.14.103.1
--webserver_interface=127.14.103.1
--webserver_port=0
--tserver_master_addrs=127.14.103.62:32917
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.1.keytab
--principal=kudu/127.14.103.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.103.20:42651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501898713699-14748-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501898713699-14748-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501898713699-14748-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:32:07.918097 19337 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:32:07.918444 19337 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:32:07.918499 19337 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:32:07.922919 19337 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:32:07.923105 19337 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:32:07.923188 19337 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.103.1
I20260812 02:32:07.929075 19337 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.103.20:42651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501898713699-14748-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501898713699-14748-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.1.keytab
--principal=kudu/127.14.103.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.103.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501898713699-14748-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.103.1
--webserver_port=0
--tserver_master_addrs=127.14.103.62:32917
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19337
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.103.1
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501898713699-14748-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:32:07.930223 19337 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:32:07.931305 19337 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:32:07.943923 19343 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:32:07.943994 19337 server_base.cc:1034] running on GCE node
W20260812 02:32:07.943928 19347 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:32:07.944473 19344 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:32:07.945001 19337 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:32:07.945798 19337 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:32:07.947084 19337 hybrid_clock.cc:648] HybridClock initialized: now 1786501927947071 us; error 37 us; skew 500 ppm
Aug 12 02:32:07 dist-test-slave-2w3w krb5kdc[19219](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501927, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:32:07.950491 19337 init.cc:377] Logged in from keytab as kudu/127.14.103.1@KRBTEST.COM (short username kudu)
I20260812 02:32:07.951462 19337 fs_manager.cc:359] Metadata directory not provided
I20260812 02:32:07.951541 19337 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:32:07.951728 19337 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:32:07.954108 19337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501898713699-14748-0/minicluster-data/ts-0/data/instance:
uuid: "a05f7e63e0cf44b299a3280e8360d34c"
format_stamp: "Formatted at 2026-08-12 02:32:07 on dist-test-slave-2w3w"
server_key: "edd1c2648587695995ce22ad6f961a28"
server_key_iv: "1b428c6392c762d2ab00daddcb5e4eb7"
server_key_version: "encryptionkey@0"
I20260812 02:32:07.954847 19337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501898713699-14748-0/minicluster-data/ts-0/wal/instance:
uuid: "a05f7e63e0cf44b299a3280e8360d34c"
format_stamp: "Formatted at 2026-08-12 02:32:07 on dist-test-slave-2w3w"
server_key: "edd1c2648587695995ce22ad6f961a28"
server_key_iv: "1b428c6392c762d2ab00daddcb5e4eb7"
server_key_version: "encryptionkey@0"
I20260812 02:32:07.959333 19337 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260812 02:32:07.962817 19353 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:32:07.964823 19337 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 02:32:07.965004 19337 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501898713699-14748-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501898713699-14748-0/minicluster-data/ts-0/wal
uuid: "a05f7e63e0cf44b299a3280e8360d34c"
format_stamp: "Formatted at 2026-08-12 02:32:07 on dist-test-slave-2w3w"
server_key: "edd1c2648587695995ce22ad6f961a28"
server_key_iv: "1b428c6392c762d2ab00daddcb5e4eb7"
server_key_version: "encryptionkey@0"
I20260812 02:32:07.965157 19337 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501898713699-14748-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501898713699-14748-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501898713699-14748-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:32:07.990084 19337 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:32:07.994815 19337 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:32:07.995076 19337 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:32:07.995769 19337 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:32:07.996734 19337 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:32:07.996826 19337 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:32:07.996901 19337 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:32:07.996944 19337 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:32:08.008221 19337 rpc_server.cc:307] RPC server started. Bound to: 127.14.103.1:33127
I20260812 02:32:08.008323 19473 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.103.1:33127 every 8 connection(s)
I20260812 02:32:08.009008 19337 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501898713699-14748-0/minicluster-data/ts-0/data/info.pb
I20260812 02:32:08.015231 14748 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbep4M4/build/debug/bin/kudu as pid 19337
I20260812 02:32:08.015394 14748 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501898713699-14748-0/minicluster-data/ts-0/wal/instance
I20260812 02:32:08.015740 14748 external_mini_cluster.cc:1428] Setting key c7fbe84eafad4373bfe4088745bc3002
Aug 12 02:32:08 dist-test-slave-2w3w krb5kdc[19219](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501927, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.1@KRBTEST.COM for kudu/127.14.103.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.14.103.2@KRBTEST.COM; defaulting to no policy
I20260812 02:32:08.028999 19309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:08.011311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.1:58807 (local address 127.14.103.62:32917)
0812 02:32:08.011530 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:08.011535 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:08.012575 (+ 1040us) server_negotiation.cc:407] Connection header received
0812 02:32:08.014059 (+ 1484us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:08.014063 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:08.014128 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:08.014217 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:32:08.016303 (+ 2086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:08.017022 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:08.019127 (+ 2105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:08.019412 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:08.019518 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:08.022844 (+ 3326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:32:08.022852 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:32:08.022857 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:32:08.022900 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:32:08.025335 (+ 2435us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:08.026178 (+ 843us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:08.026183 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:08.026186 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:08.026298 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:08.027305 (+ 1007us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:08.027310 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:08.027312 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:08.027616 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:32:08.027749 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0812 02:32:08.028704 (+ 955us) server_negotiation.cc:299] Negotiation successful
0812 02:32:08.028851 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260812 02:32:08.029850 19477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:08.011701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:32917 (local address 127.14.103.1:58807)
0812 02:32:08.012367 (+ 666us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:08.012423 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0812 02:32:08.013753 (+ 1330us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:32:08.014434 (+ 681us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:32:08.014442 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:32:08.015040 (+ 598us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:32:08.015858 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:08.015884 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:08.017248 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:08.017254 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:08.018897 (+ 1643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:08.018916 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:08.019615 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:08.019622 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:08.019908 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:08.020650 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:32:08.020669 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:32:08.022490 (+ 1821us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:32:08.025557 (+ 3067us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:32:08.025565 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:32:08.025583 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:32:08.026012 (+ 429us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:32:08.026488 (+ 476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:32:08.026491 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:32:08.026493 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:32:08.026957 (+ 464us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:32:08.027853 (+ 896us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:32:08.027860 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:32:08.028196 (+ 336us) client_negotiation.cc:769] Sending connection context
0812 02:32:08.028438 (+ 242us) client_negotiation.cc:240] Negotiation successful
0812 02:32:08.028773 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":200,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.103.2@KRBTEST.COM" created.
I20260812 02:32:08.032220 19475 heartbeater.cc:347] Connected to a master server at 127.14.103.62:32917
I20260812 02:32:08.032629 19475 heartbeater.cc:464] Registering TS with master...
I20260812 02:32:08.033362 19475 heartbeater.cc:511] Master 127.14.103.62:32917 requested a full tablet report, sending...
I20260812 02:32:08.035406 19268 ts_manager.cc:194] Registered new tserver with Master: a05f7e63e0cf44b299a3280e8360d34c (127.14.103.1:33127)
I20260812 02:32:08.037117 19268 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.103.1@KRBTEST.COM'} at 127.14.103.1:58807
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.103.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.2.keytab.
Entry for principal kudu/127.14.103.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.2.keytab.
I20260812 02:32:08.056962 14748 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbep4M4/build/debug/bin/kudu
/tmp/dist-test-taskbep4M4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501898713699-14748-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501898713699-14748-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501898713699-14748-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501898713699-14748-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.103.2:0
--local_ip_for_outbound_sockets=127.14.103.2
--webserver_interface=127.14.103.2
--webserver_port=0
--tserver_master_addrs=127.14.103.62:32917
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.2.keytab
--principal=kudu/127.14.103.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.103.20:42651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501898713699-14748-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501898713699-14748-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501898713699-14748-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:32:08.193351 19481 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:32:08.193738 19481 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:32:08.193794 19481 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:32:08.198572 19481 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:32:08.198676 19481 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:32:08.198746 19481 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.103.2
I20260812 02:32:08.204898 19481 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.103.20:42651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501898713699-14748-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501898713699-14748-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.2.keytab
--principal=kudu/127.14.103.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.103.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501898713699-14748-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.103.2
--webserver_port=0
--tserver_master_addrs=127.14.103.62:32917
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19481
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.103.2
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501898713699-14748-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:32:08.206148 19481 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:32:08.207274 19481 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:32:08.218885 19488 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:32:08.218779 19490 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:32:08.218782 19487 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:32:08.219219 19481 server_base.cc:1034] running on GCE node
I20260812 02:32:08.219944 19481 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:32:08.220690 19481 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:32:08.221889 19481 hybrid_clock.cc:648] HybridClock initialized: now 1786501928221829 us; error 90 us; skew 500 ppm
Aug 12 02:32:08 dist-test-slave-2w3w krb5kdc[19219](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501928, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:32:08.226310 19481 init.cc:377] Logged in from keytab as kudu/127.14.103.2@KRBTEST.COM (short username kudu)
I20260812 02:32:08.227365 19481 fs_manager.cc:359] Metadata directory not provided
I20260812 02:32:08.227478 19481 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:32:08.227728 19481 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:32:08.230057 19481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501898713699-14748-0/minicluster-data/ts-1/data/instance:
uuid: "3cc8386158ac40d38bca578791f1770d"
format_stamp: "Formatted at 2026-08-12 02:32:08 on dist-test-slave-2w3w"
server_key: "179c50fba36384ff8c0418724d34d544"
server_key_iv: "980db4fcec7fe5bb1be223188097a07b"
server_key_version: "encryptionkey@0"
I20260812 02:32:08.230726 19481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501898713699-14748-0/minicluster-data/ts-1/wal/instance:
uuid: "3cc8386158ac40d38bca578791f1770d"
format_stamp: "Formatted at 2026-08-12 02:32:08 on dist-test-slave-2w3w"
server_key: "179c50fba36384ff8c0418724d34d544"
server_key_iv: "980db4fcec7fe5bb1be223188097a07b"
server_key_version: "encryptionkey@0"
I20260812 02:32:08.235481 19481 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.003s
I20260812 02:32:08.238642 19497 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:32:08.240433 19481 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260812 02:32:08.240847 19481 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501898713699-14748-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501898713699-14748-0/minicluster-data/ts-1/wal
uuid: "3cc8386158ac40d38bca578791f1770d"
format_stamp: "Formatted at 2026-08-12 02:32:08 on dist-test-slave-2w3w"
server_key: "179c50fba36384ff8c0418724d34d544"
server_key_iv: "980db4fcec7fe5bb1be223188097a07b"
server_key_version: "encryptionkey@0"
I20260812 02:32:08.241029 19481 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501898713699-14748-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501898713699-14748-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501898713699-14748-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:32:08.259128 19481 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:32:08.263355 19481 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:32:08.263617 19481 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:32:08.264421 19481 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:32:08.265719 19481 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:32:08.265818 19481 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:32:08.265906 19481 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:32:08.265952 19481 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:32:08.275990 19481 rpc_server.cc:307] RPC server started. Bound to: 127.14.103.2:36133
I20260812 02:32:08.276109 19622 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.103.2:36133 every 8 connection(s)
I20260812 02:32:08.276687 19481 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501898713699-14748-0/minicluster-data/ts-1/data/info.pb
I20260812 02:32:08.277034 14748 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbep4M4/build/debug/bin/kudu as pid 19481
I20260812 02:32:08.277155 14748 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501898713699-14748-0/minicluster-data/ts-1/wal/instance
I20260812 02:32:08.277486 14748 external_mini_cluster.cc:1428] Setting key 3db67ad18949aed5a62e3258671eff6e
Aug 12 02:32:08 dist-test-slave-2w3w krb5kdc[19219](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501928, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.2@KRBTEST.COM for kudu/127.14.103.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.14.103.3@KRBTEST.COM; defaulting to no policy
I20260812 02:32:08.294942 19309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:08.278932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.2:53985 (local address 127.14.103.62:32917)
0812 02:32:08.279103 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:08.279107 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:08.280036 (+ 929us) server_negotiation.cc:407] Connection header received
0812 02:32:08.281596 (+ 1560us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:08.281600 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:08.281658 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:08.281747 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:32:08.283047 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:08.283740 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:08.285139 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:08.285395 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:08.285515 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:08.289060 (+ 3545us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:32:08.289063 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:32:08.289065 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:32:08.289095 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:32:08.291394 (+ 2299us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:08.292096 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:08.292102 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:08.292104 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:08.292211 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:08.292738 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:08.292775 (+ 37us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:08.292776 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:08.293124 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:32:08.293603 (+ 479us) server_negotiation.cc:1035] Waiting for connection context
0812 02:32:08.294529 (+ 926us) server_negotiation.cc:299] Negotiation successful
0812 02:32:08.294759 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.103.3@KRBTEST.COM" created.
I20260812 02:32:08.295948 19626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:08.279383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:32917 (local address 127.14.103.2:53985)
0812 02:32:08.279863 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:08.279908 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0812 02:32:08.281327 (+ 1419us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:32:08.281950 (+ 623us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:32:08.281957 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:32:08.282336 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:32:08.282860 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:08.282874 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:08.283938 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:08.283943 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:08.284969 (+ 1026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:08.284984 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:08.285587 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:08.285591 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:08.285780 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:08.286559 (+ 779us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:32:08.286578 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:32:08.288815 (+ 2237us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:32:08.291568 (+ 2753us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:32:08.291583 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:32:08.291601 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:32:08.291962 (+ 361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:32:08.292320 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:32:08.292323 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:32:08.292326 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:32:08.292596 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:32:08.293613 (+ 1017us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:32:08.293625 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:32:08.294169 (+ 544us) client_negotiation.cc:769] Sending connection context
0812 02:32:08.294417 (+ 248us) client_negotiation.cc:240] Negotiation successful
0812 02:32:08.294854 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":121,"threads_started":1}
I20260812 02:32:08.297478 19624 heartbeater.cc:347] Connected to a master server at 127.14.103.62:32917
I20260812 02:32:08.297798 19624 heartbeater.cc:464] Registering TS with master...
I20260812 02:32:08.298491 19624 heartbeater.cc:511] Master 127.14.103.62:32917 requested a full tablet report, sending...
I20260812 02:32:08.299866 19268 ts_manager.cc:194] Registered new tserver with Master: 3cc8386158ac40d38bca578791f1770d (127.14.103.2:36133)
I20260812 02:32:08.300581 19268 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.103.2@KRBTEST.COM'} at 127.14.103.2:53985
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.103.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.3.keytab.
Entry for principal kudu/127.14.103.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.3.keytab.
I20260812 02:32:08.320483 14748 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbep4M4/build/debug/bin/kudu
/tmp/dist-test-taskbep4M4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501898713699-14748-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501898713699-14748-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501898713699-14748-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501898713699-14748-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.103.3:0
--local_ip_for_outbound_sockets=127.14.103.3
--webserver_interface=127.14.103.3
--webserver_port=0
--tserver_master_addrs=127.14.103.62:32917
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.3.keytab
--principal=kudu/127.14.103.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.103.20:42651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501898713699-14748-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501898713699-14748-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501898713699-14748-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:32:08.476963 19629 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:32:08.477319 19629 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:32:08.477406 19629 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:32:08.481894 19629 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:32:08.482004 19629 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:32:08.482105 19629 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.103.3
I20260812 02:32:08.488442 19629 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.103.20:42651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501898713699-14748-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501898713699-14748-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.3.keytab
--principal=kudu/127.14.103.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.103.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501898713699-14748-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.103.3
--webserver_port=0
--tserver_master_addrs=127.14.103.62:32917
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19629
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.103.3
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501898713699-14748-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:32:08.489923 19629 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:32:08.491307 19629 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:32:08.503187 19636 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:32:08.503192 19637 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:32:08.503676 19639 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:32:08.504051 19629 server_base.cc:1034] running on GCE node
I20260812 02:32:08.504566 19629 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:32:08.505329 19629 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:32:08.506541 19629 hybrid_clock.cc:648] HybridClock initialized: now 1786501928506455 us; error 102 us; skew 500 ppm
Aug 12 02:32:08 dist-test-slave-2w3w krb5kdc[19219](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501928, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:32:08.511394 19629 init.cc:377] Logged in from keytab as kudu/127.14.103.3@KRBTEST.COM (short username kudu)
I20260812 02:32:08.512283 19629 fs_manager.cc:359] Metadata directory not provided
I20260812 02:32:08.512387 19629 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:32:08.512637 19629 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:32:08.514938 19629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501898713699-14748-0/minicluster-data/ts-2/data/instance:
uuid: "b84d4904c86d4111b8ca63a0c0fe3d59"
format_stamp: "Formatted at 2026-08-12 02:32:08 on dist-test-slave-2w3w"
server_key: "af5d7b0acc32d33518a9e4e240eb75f5"
server_key_iv: "d584f490a06e23176afd1804a28610e5"
server_key_version: "encryptionkey@0"
I20260812 02:32:08.515659 19629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501898713699-14748-0/minicluster-data/ts-2/wal/instance:
uuid: "b84d4904c86d4111b8ca63a0c0fe3d59"
format_stamp: "Formatted at 2026-08-12 02:32:08 on dist-test-slave-2w3w"
server_key: "af5d7b0acc32d33518a9e4e240eb75f5"
server_key_iv: "d584f490a06e23176afd1804a28610e5"
server_key_version: "encryptionkey@0"
I20260812 02:32:08.520152 19629 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260812 02:32:08.523115 19645 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:32:08.524598 19629 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260812 02:32:08.524771 19629 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501898713699-14748-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501898713699-14748-0/minicluster-data/ts-2/wal
uuid: "b84d4904c86d4111b8ca63a0c0fe3d59"
format_stamp: "Formatted at 2026-08-12 02:32:08 on dist-test-slave-2w3w"
server_key: "af5d7b0acc32d33518a9e4e240eb75f5"
server_key_iv: "d584f490a06e23176afd1804a28610e5"
server_key_version: "encryptionkey@0"
I20260812 02:32:08.524912 19629 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501898713699-14748-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501898713699-14748-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501898713699-14748-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:32:08.551931 19629 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:32:08.555815 19629 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:32:08.556047 19629 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:32:08.556694 19629 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:32:08.557644 19629 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:32:08.557717 19629 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:32:08.557799 19629 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:32:08.557848 19629 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:32:08.567798 19629 rpc_server.cc:307] RPC server started. Bound to: 127.14.103.3:33165
I20260812 02:32:08.567816 19772 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.103.3:33165 every 8 connection(s)
I20260812 02:32:08.568709 19629 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501898713699-14748-0/minicluster-data/ts-2/data/info.pb
I20260812 02:32:08.570302 14748 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbep4M4/build/debug/bin/kudu as pid 19629
I20260812 02:32:08.570482 14748 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501898713699-14748-0/minicluster-data/ts-2/wal/instance
I20260812 02:32:08.570776 14748 external_mini_cluster.cc:1428] Setting key 85775120e618f91f3283cec86ac15fdf
Aug 12 02:32:08 dist-test-slave-2w3w krb5kdc[19219](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501928, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.3@KRBTEST.COM for kudu/127.14.103.62@KRBTEST.COM
I20260812 02:32:08.584376 19309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:08.570931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.3:40309 (local address 127.14.103.62:32917)
0812 02:32:08.571102 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:08.571108 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:08.572251 (+ 1143us) server_negotiation.cc:407] Connection header received
0812 02:32:08.573717 (+ 1466us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:08.573720 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:08.573781 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:08.573865 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:32:08.575195 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:08.575749 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:08.576945 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:08.577174 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:08.577289 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:08.580079 (+ 2790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:32:08.580083 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:32:08.580085 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:32:08.580112 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:32:08.581696 (+ 1584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:08.582360 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:08.582364 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:08.582370 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:08.582458 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:08.582861 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:08.582870 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:08.582871 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:08.583161 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:32:08.583299 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0812 02:32:08.584000 (+ 701us) server_negotiation.cc:299] Negotiation successful
0812 02:32:08.584182 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260812 02:32:08.585556 19777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:08.571425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:32917 (local address 127.14.103.3:40309)
0812 02:32:08.572083 (+ 658us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:08.572144 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0812 02:32:08.573514 (+ 1370us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:32:08.574068 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:32:08.574075 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:32:08.574466 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:32:08.575051 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:08.575069 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:08.575885 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:08.575888 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:08.576800 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:08.576810 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:08.577337 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:08.577341 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:08.577541 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:08.578248 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:32:08.578265 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:32:08.579872 (+ 1607us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:32:08.581854 (+ 1982us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:32:08.581863 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:32:08.581886 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:32:08.582228 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:32:08.582562 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:32:08.582566 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:32:08.582568 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:32:08.582734 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:32:08.583297 (+ 563us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:32:08.583305 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:32:08.583621 (+ 316us) client_negotiation.cc:769] Sending connection context
0812 02:32:08.583850 (+ 229us) client_negotiation.cc:240] Negotiation successful
0812 02:32:08.584271 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":171,"threads_started":1}
I20260812 02:32:08.587376 19774 heartbeater.cc:347] Connected to a master server at 127.14.103.62:32917
I20260812 02:32:08.587653 19774 heartbeater.cc:464] Registering TS with master...
I20260812 02:32:08.588392 19774 heartbeater.cc:511] Master 127.14.103.62:32917 requested a full tablet report, sending...
I20260812 02:32:08.589803 19268 ts_manager.cc:194] Registered new tserver with Master: b84d4904c86d4111b8ca63a0c0fe3d59 (127.14.103.3:33165)
I20260812 02:32:08.590440 19268 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.103.3@KRBTEST.COM'} at 127.14.103.3:40309
I20260812 02:32:08.599136 14748 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 02:32:08.617408 19309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:08.603915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39008 (local address 127.14.103.62:32917)
0812 02:32:08.604099 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:08.604105 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:08.604690 (+ 585us) server_negotiation.cc:407] Connection header received
0812 02:32:08.604903 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:08.604906 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:08.604959 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:08.605051 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:32:08.606950 (+ 1899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:08.607784 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:08.610097 (+ 2313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:08.610468 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:08.610590 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:08.612764 (+ 2174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:32:08.612768 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:32:08.612770 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:32:08.612805 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:32:08.614765 (+ 1960us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:08.615379 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:08.615382 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:08.615383 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:08.615479 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:08.615935 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:08.615940 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:08.615942 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:08.616422 (+ 480us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:32:08.616606 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0812 02:32:08.616991 (+ 385us) server_negotiation.cc:299] Negotiation successful
0812 02:32:08.617191 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260812 02:32:08.621870 19268 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39008:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 02:32:08.625802 19268 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 02:32:08.646782 19787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:08.638017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.1:33127 (local address 127.0.0.1:42638)
0812 02:32:08.638942 (+ 925us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:08.639037 (+ 95us) client_negotiation.cc:174] Beginning negotiation
0812 02:32:08.639192 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:32:08.641084 (+ 1892us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:32:08.641090 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:32:08.641129 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:32:08.641347 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:08.641355 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:08.643449 (+ 2094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:08.643453 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:08.644841 (+ 1388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:08.644853 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:08.646167 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:08.646171 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:08.646303 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:08.646351 (+ 48us) client_negotiation.cc:769] Sending connection context
0812 02:32:08.646450 (+ 99us) client_negotiation.cc:240] Negotiation successful
0812 02:32:08.646599 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":735,"mutex_wait_us":19,"spinlock_wait_cycles":3584,"thread_start_us":111,"threads_started":1}
I20260812 02:32:08.647256 19790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:08.638767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44932 (local address 127.14.103.3:33165)
0812 02:32:08.639904 (+ 1137us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:08.639916 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:08.640183 (+ 267us) server_negotiation.cc:407] Connection header received
0812 02:32:08.640420 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:08.640428 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:08.640676 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:08.640827 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:32:08.641616 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:08.642843 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:08.645178 (+ 2335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:08.646175 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:08.646300 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:08.646467 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0812 02:32:08.646903 (+ 436us) server_negotiation.cc:299] Negotiation successful
0812 02:32:08.647072 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1005,"thread_start_us":318,"threads_started":1}
I20260812 02:32:08.647908 19786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:08.639266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.2:36133 (local address 127.0.0.1:44194)
0812 02:32:08.640043 (+ 777us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:08.640058 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 02:32:08.640145 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:32:08.641346 (+ 1201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:32:08.641349 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:32:08.641366 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:32:08.641469 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:08.641475 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:08.643099 (+ 1624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:08.643102 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:08.644695 (+ 1593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:08.644707 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:08.647459 (+ 2752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:08.647464 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:08.647574 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:08.647606 (+ 32us) client_negotiation.cc:769] Sending connection context
0812 02:32:08.647669 (+ 63us) client_negotiation.cc:240] Negotiation successful
0812 02:32:08.647751 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":709,"mutex_wait_us":17,"thread_start_us":131,"threads_started":1}
I20260812 02:32:08.648334 19789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:08.638234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42638 (local address 127.14.103.1:33127)
0812 02:32:08.640419 (+ 2185us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:08.640426 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:08.640450 (+ 24us) server_negotiation.cc:407] Connection header received
0812 02:32:08.640530 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:08.640535 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:08.640901 (+ 366us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:08.641323 (+ 422us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:32:08.641468 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:08.643318 (+ 1850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:08.645026 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:08.646008 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:08.647562 (+ 1554us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:08.647728 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0812 02:32:08.647786 (+ 58us) server_negotiation.cc:299] Negotiation successful
0812 02:32:08.647899 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2068,"thread_start_us":127,"threads_started":1}
I20260812 02:32:08.647909 19791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:08.638121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.3:33165 (local address 127.0.0.1:44932)
0812 02:32:08.640070 (+ 1949us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:08.640085 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 02:32:08.640261 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:32:08.641082 (+ 821us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:32:08.641088 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:32:08.641128 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:32:08.641479 (+ 351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:08.641483 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:08.643078 (+ 1595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:08.643085 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:08.645031 (+ 1946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:08.645038 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:08.646538 (+ 1500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:08.646541 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:08.646640 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:08.646664 (+ 24us) client_negotiation.cc:769] Sending connection context
0812 02:32:08.646733 (+ 69us) client_negotiation.cc:240] Negotiation successful
0812 02:32:08.646811 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1810,"mutex_wait_us":46,"spinlock_wait_cycles":8320,"thread_start_us":95,"threads_started":1}
I20260812 02:32:08.649102 19792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:08.639899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44194 (local address 127.14.103.2:36133)
0812 02:32:08.640887 (+ 988us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:08.640898 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:08.640920 (+ 22us) server_negotiation.cc:407] Connection header received
0812 02:32:08.640999 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:08.641007 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:08.641186 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:08.641637 (+ 451us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:32:08.641803 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:08.642928 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:08.645215 (+ 2287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:08.646173 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:08.648134 (+ 1961us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:08.648494 (+ 360us) server_negotiation.cc:1035] Waiting for connection context
0812 02:32:08.648766 (+ 272us) server_negotiation.cc:299] Negotiation successful
0812 02:32:08.648917 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":883,"thread_start_us":118,"threads_started":1}
I20260812 02:32:08.652369 19697 tablet_service.cc:1467] Processing CreateTablet for tablet a01e420de0e3438e9080917aca8dd75c (DEFAULT_TABLE table=test-table [id=07347f491c784960bae27977b0b08390]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:32:08.652813 19403 tablet_service.cc:1467] Processing CreateTablet for tablet a01e420de0e3438e9080917aca8dd75c (DEFAULT_TABLE table=test-table [id=07347f491c784960bae27977b0b08390]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:32:08.652841 19548 tablet_service.cc:1467] Processing CreateTablet for tablet a01e420de0e3438e9080917aca8dd75c (DEFAULT_TABLE table=test-table [id=07347f491c784960bae27977b0b08390]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:32:08.653620 19697 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a01e420de0e3438e9080917aca8dd75c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:32:08.653911 19403 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a01e420de0e3438e9080917aca8dd75c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:32:08.653911 19548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a01e420de0e3438e9080917aca8dd75c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:32:08.663281 19796 tablet_bootstrap.cc:492] T a01e420de0e3438e9080917aca8dd75c P 3cc8386158ac40d38bca578791f1770d: Bootstrap starting.
I20260812 02:32:08.663980 19793 tablet_bootstrap.cc:492] T a01e420de0e3438e9080917aca8dd75c P b84d4904c86d4111b8ca63a0c0fe3d59: Bootstrap starting.
I20260812 02:32:08.664522 19794 tablet_bootstrap.cc:492] T a01e420de0e3438e9080917aca8dd75c P a05f7e63e0cf44b299a3280e8360d34c: Bootstrap starting.
I20260812 02:32:08.666425 19793 tablet_bootstrap.cc:654] T a01e420de0e3438e9080917aca8dd75c P b84d4904c86d4111b8ca63a0c0fe3d59: Neither blocks nor log segments found. Creating new log.
I20260812 02:32:08.666589 19796 tablet_bootstrap.cc:654] T a01e420de0e3438e9080917aca8dd75c P 3cc8386158ac40d38bca578791f1770d: Neither blocks nor log segments found. Creating new log.
I20260812 02:32:08.667096 19794 tablet_bootstrap.cc:654] T a01e420de0e3438e9080917aca8dd75c P a05f7e63e0cf44b299a3280e8360d34c: Neither blocks nor log segments found. Creating new log.
I20260812 02:32:08.667593 19796 log.cc:824] T a01e420de0e3438e9080917aca8dd75c P 3cc8386158ac40d38bca578791f1770d: Log is configured to *not* fsync() on all Append() calls
I20260812 02:32:08.668015 19794 log.cc:824] T a01e420de0e3438e9080917aca8dd75c P a05f7e63e0cf44b299a3280e8360d34c: Log is configured to *not* fsync() on all Append() calls
I20260812 02:32:08.668015 19793 log.cc:824] T a01e420de0e3438e9080917aca8dd75c P b84d4904c86d4111b8ca63a0c0fe3d59: Log is configured to *not* fsync() on all Append() calls
I20260812 02:32:08.670441 19796 tablet_bootstrap.cc:492] T a01e420de0e3438e9080917aca8dd75c P 3cc8386158ac40d38bca578791f1770d: No bootstrap required, opened a new log
I20260812 02:32:08.670493 19794 tablet_bootstrap.cc:492] T a01e420de0e3438e9080917aca8dd75c P a05f7e63e0cf44b299a3280e8360d34c: No bootstrap required, opened a new log
I20260812 02:32:08.670481 19793 tablet_bootstrap.cc:492] T a01e420de0e3438e9080917aca8dd75c P b84d4904c86d4111b8ca63a0c0fe3d59: No bootstrap required, opened a new log
I20260812 02:32:08.670697 19794 ts_tablet_manager.cc:1397] T a01e420de0e3438e9080917aca8dd75c P a05f7e63e0cf44b299a3280e8360d34c: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260812 02:32:08.670697 19793 ts_tablet_manager.cc:1397] T a01e420de0e3438e9080917aca8dd75c P b84d4904c86d4111b8ca63a0c0fe3d59: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20260812 02:32:08.670701 19796 ts_tablet_manager.cc:1397] T a01e420de0e3438e9080917aca8dd75c P 3cc8386158ac40d38bca578791f1770d: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260812 02:32:08.673895 19796 raft_consensus.cc:348] T a01e420de0e3438e9080917aca8dd75c P 3cc8386158ac40d38bca578791f1770d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a05f7e63e0cf44b299a3280e8360d34c" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 33127 } } peers { permanent_uuid: "b84d4904c86d4111b8ca63a0c0fe3d59" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 33165 } } peers { permanent_uuid: "3cc8386158ac40d38bca578791f1770d" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 36133 } }
I20260812 02:32:08.674232 19796 raft_consensus.cc:374] T a01e420de0e3438e9080917aca8dd75c P 3cc8386158ac40d38bca578791f1770d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:32:08.674336 19796 raft_consensus.cc:729] T a01e420de0e3438e9080917aca8dd75c P 3cc8386158ac40d38bca578791f1770d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3cc8386158ac40d38bca578791f1770d, State: Initialized, Role: FOLLOWER
I20260812 02:32:08.674883 19796 consensus_queue.cc:260] T a01e420de0e3438e9080917aca8dd75c P 3cc8386158ac40d38bca578791f1770d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a05f7e63e0cf44b299a3280e8360d34c" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 33127 } } peers { permanent_uuid: "b84d4904c86d4111b8ca63a0c0fe3d59" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 33165 } } peers { permanent_uuid: "3cc8386158ac40d38bca578791f1770d" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 36133 } }
I20260812 02:32:08.675030 19793 raft_consensus.cc:348] T a01e420de0e3438e9080917aca8dd75c P b84d4904c86d4111b8ca63a0c0fe3d59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a05f7e63e0cf44b299a3280e8360d34c" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 33127 } } peers { permanent_uuid: "b84d4904c86d4111b8ca63a0c0fe3d59" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 33165 } } peers { permanent_uuid: "3cc8386158ac40d38bca578791f1770d" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 36133 } }
I20260812 02:32:08.675393 19793 raft_consensus.cc:374] T a01e420de0e3438e9080917aca8dd75c P b84d4904c86d4111b8ca63a0c0fe3d59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:32:08.675490 19793 raft_consensus.cc:729] T a01e420de0e3438e9080917aca8dd75c P b84d4904c86d4111b8ca63a0c0fe3d59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b84d4904c86d4111b8ca63a0c0fe3d59, State: Initialized, Role: FOLLOWER
I20260812 02:32:08.675460 19794 raft_consensus.cc:348] T a01e420de0e3438e9080917aca8dd75c P a05f7e63e0cf44b299a3280e8360d34c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a05f7e63e0cf44b299a3280e8360d34c" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 33127 } } peers { permanent_uuid: "b84d4904c86d4111b8ca63a0c0fe3d59" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 33165 } } peers { permanent_uuid: "3cc8386158ac40d38bca578791f1770d" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 36133 } }
I20260812 02:32:08.675868 19794 raft_consensus.cc:374] T a01e420de0e3438e9080917aca8dd75c P a05f7e63e0cf44b299a3280e8360d34c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:32:08.675971 19794 raft_consensus.cc:729] T a01e420de0e3438e9080917aca8dd75c P a05f7e63e0cf44b299a3280e8360d34c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a05f7e63e0cf44b299a3280e8360d34c, State: Initialized, Role: FOLLOWER
I20260812 02:32:08.676038 19793 consensus_queue.cc:260] T a01e420de0e3438e9080917aca8dd75c P b84d4904c86d4111b8ca63a0c0fe3d59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a05f7e63e0cf44b299a3280e8360d34c" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 33127 } } peers { permanent_uuid: "b84d4904c86d4111b8ca63a0c0fe3d59" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 33165 } } peers { permanent_uuid: "3cc8386158ac40d38bca578791f1770d" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 36133 } }
I20260812 02:32:08.676273 19624 heartbeater.cc:503] Master 127.14.103.62:32917 was elected leader, sending a full tablet report...
I20260812 02:32:08.677455 19774 heartbeater.cc:503] Master 127.14.103.62:32917 was elected leader, sending a full tablet report...
I20260812 02:32:08.677523 19796 ts_tablet_manager.cc:1428] T a01e420de0e3438e9080917aca8dd75c P 3cc8386158ac40d38bca578791f1770d: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260812 02:32:08.677724 19793 ts_tablet_manager.cc:1428] T a01e420de0e3438e9080917aca8dd75c P b84d4904c86d4111b8ca63a0c0fe3d59: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260812 02:32:08.678648 19794 consensus_queue.cc:260] T a01e420de0e3438e9080917aca8dd75c P a05f7e63e0cf44b299a3280e8360d34c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a05f7e63e0cf44b299a3280e8360d34c" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 33127 } } peers { permanent_uuid: "b84d4904c86d4111b8ca63a0c0fe3d59" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 33165 } } peers { permanent_uuid: "3cc8386158ac40d38bca578791f1770d" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 36133 } }
I20260812 02:32:08.679787 19475 heartbeater.cc:503] Master 127.14.103.62:32917 was elected leader, sending a full tablet report...
I20260812 02:32:08.679965 19794 ts_tablet_manager.cc:1428] T a01e420de0e3438e9080917aca8dd75c P a05f7e63e0cf44b299a3280e8360d34c: Time spent starting tablet: real 0.009s user 0.007s sys 0.000s
W20260812 02:32:08.760667 19476 tablet.cc:2369] T a01e420de0e3438e9080917aca8dd75c P a05f7e63e0cf44b299a3280e8360d34c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 02:32:08.777978 19625 tablet.cc:2369] T a01e420de0e3438e9080917aca8dd75c P 3cc8386158ac40d38bca578791f1770d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 02:32:08.819855 19776 tablet.cc:2369] T a01e420de0e3438e9080917aca8dd75c P b84d4904c86d4111b8ca63a0c0fe3d59: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 02:32:08.881096 19800 raft_consensus.cc:482] T a01e420de0e3438e9080917aca8dd75c P 3cc8386158ac40d38bca578791f1770d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 02:32:08.881410 19800 raft_consensus.cc:504] T a01e420de0e3438e9080917aca8dd75c P 3cc8386158ac40d38bca578791f1770d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a05f7e63e0cf44b299a3280e8360d34c" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 33127 } } peers { permanent_uuid: "b84d4904c86d4111b8ca63a0c0fe3d59" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 33165 } } peers { permanent_uuid: "3cc8386158ac40d38bca578791f1770d" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 36133 } }
I20260812 02:32:08.882592 19800 leader_election.cc:288] T a01e420de0e3438e9080917aca8dd75c P 3cc8386158ac40d38bca578791f1770d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a05f7e63e0cf44b299a3280e8360d34c (127.14.103.1:33127), b84d4904c86d4111b8ca63a0c0fe3d59 (127.14.103.3:33165)
I20260812 02:32:08.888990 19789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:08.883147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.2:40037 (local address 127.14.103.1:33127)
0812 02:32:08.883298 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:08.883304 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:08.883507 (+ 203us) server_negotiation.cc:407] Connection header received
0812 02:32:08.883618 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:08.883621 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:08.883679 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:08.883788 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:32:08.884392 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:08.885277 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:08.887417 (+ 2140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:08.888142 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:08.888306 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:08.888399 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0812 02:32:08.888734 (+ 335us) server_negotiation.cc:299] Negotiation successful
0812 02:32:08.888832 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260812 02:32:08.888990 19790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:08.883155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.2:60065 (local address 127.14.103.3:33165)
0812 02:32:08.883298 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:08.883304 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:08.883566 (+ 262us) server_negotiation.cc:407] Connection header received
0812 02:32:08.883626 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:08.883630 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:08.883678 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:08.883788 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:32:08.884389 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:08.885295 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:08.887417 (+ 2122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:08.888149 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:08.888306 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:08.888370 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0812 02:32:08.888734 (+ 364us) server_negotiation.cc:299] Negotiation successful
0812 02:32:08.888860 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260812 02:32:08.889040 19804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:08.882920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.3:33165 (local address 127.14.103.2:60065)
0812 02:32:08.883463 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:08.883476 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 02:32:08.883550 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:32:08.883880 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:32:08.883884 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:32:08.883934 (+ 50us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:32:08.884192 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:08.884203 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:08.885472 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:08.885476 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:08.887239 (+ 1763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:08.887254 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:08.888329 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:08.888359 (+ 30us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:08.888508 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:08.888576 (+ 68us) client_negotiation.cc:769] Sending connection context
0812 02:32:08.888671 (+ 95us) client_negotiation.cc:240] Negotiation successful
0812 02:32:08.888830 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476,"mutex_wait_us":329,"spinlock_wait_cycles":29952,"thread_start_us":81,"threads_started":1}
I20260812 02:32:08.889042 19803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:08.882920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.1:33127 (local address 127.14.103.2:40037)
0812 02:32:08.883373 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:08.883390 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 02:32:08.883522 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:32:08.883880 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:32:08.883884 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:32:08.883935 (+ 51us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:32:08.884189 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:08.884203 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:08.885472 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:08.885476 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:08.887239 (+ 1763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:08.887254 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:08.888329 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:08.888334 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:08.888482 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:08.888576 (+ 94us) client_negotiation.cc:769] Sending connection context
0812 02:32:08.888671 (+ 95us) client_negotiation.cc:240] Negotiation successful
0812 02:32:08.888871 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":345,"mutex_wait_us":352,"spinlock_wait_cycles":9984,"thread_start_us":116,"threads_started":1}
I20260812 02:32:08.889909 19423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a01e420de0e3438e9080917aca8dd75c" candidate_uuid: "3cc8386158ac40d38bca578791f1770d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a05f7e63e0cf44b299a3280e8360d34c" is_pre_election: true
I20260812 02:32:08.889909 19722 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a01e420de0e3438e9080917aca8dd75c" candidate_uuid: "3cc8386158ac40d38bca578791f1770d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b84d4904c86d4111b8ca63a0c0fe3d59" is_pre_election: true
I20260812 02:32:08.890277 19423 raft_consensus.cc:2454] T a01e420de0e3438e9080917aca8dd75c P a05f7e63e0cf44b299a3280e8360d34c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3cc8386158ac40d38bca578791f1770d in term 0.
I20260812 02:32:08.890277 19722 raft_consensus.cc:2454] T a01e420de0e3438e9080917aca8dd75c P b84d4904c86d4111b8ca63a0c0fe3d59 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3cc8386158ac40d38bca578791f1770d in term 0.
I20260812 02:32:08.890951 19501 leader_election.cc:302] T a01e420de0e3438e9080917aca8dd75c P 3cc8386158ac40d38bca578791f1770d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 3cc8386158ac40d38bca578791f1770d, a05f7e63e0cf44b299a3280e8360d34c, b84d4904c86d4111b8ca63a0c0fe3d59; no voters:
I20260812 02:32:08.891350 19800 raft_consensus.cc:2781] T a01e420de0e3438e9080917aca8dd75c P 3cc8386158ac40d38bca578791f1770d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 02:32:08.891446 19800 raft_consensus.cc:482] T a01e420de0e3438e9080917aca8dd75c P 3cc8386158ac40d38bca578791f1770d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 02:32:08.891500 19800 raft_consensus.cc:3037] T a01e420de0e3438e9080917aca8dd75c P 3cc8386158ac40d38bca578791f1770d [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:32:08.892652 19800 raft_consensus.cc:504] T a01e420de0e3438e9080917aca8dd75c P 3cc8386158ac40d38bca578791f1770d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a05f7e63e0cf44b299a3280e8360d34c" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 33127 } } peers { permanent_uuid: "b84d4904c86d4111b8ca63a0c0fe3d59" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 33165 } } peers { permanent_uuid: "3cc8386158ac40d38bca578791f1770d" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 36133 } }
I20260812 02:32:08.893071 19800 leader_election.cc:288] T a01e420de0e3438e9080917aca8dd75c P 3cc8386158ac40d38bca578791f1770d [CANDIDATE]: Term 1 election: Requested vote from peers a05f7e63e0cf44b299a3280e8360d34c (127.14.103.1:33127), b84d4904c86d4111b8ca63a0c0fe3d59 (127.14.103.3:33165)
I20260812 02:32:08.893623 19722 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a01e420de0e3438e9080917aca8dd75c" candidate_uuid: "3cc8386158ac40d38bca578791f1770d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b84d4904c86d4111b8ca63a0c0fe3d59"
I20260812 02:32:08.893785 19722 raft_consensus.cc:3037] T a01e420de0e3438e9080917aca8dd75c P b84d4904c86d4111b8ca63a0c0fe3d59 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:32:08.893788 19423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a01e420de0e3438e9080917aca8dd75c" candidate_uuid: "3cc8386158ac40d38bca578791f1770d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a05f7e63e0cf44b299a3280e8360d34c"
I20260812 02:32:08.893980 19423 raft_consensus.cc:3037] T a01e420de0e3438e9080917aca8dd75c P a05f7e63e0cf44b299a3280e8360d34c [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:32:08.895124 19722 raft_consensus.cc:2454] T a01e420de0e3438e9080917aca8dd75c P b84d4904c86d4111b8ca63a0c0fe3d59 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3cc8386158ac40d38bca578791f1770d in term 1.
I20260812 02:32:08.895444 19423 raft_consensus.cc:2454] T a01e420de0e3438e9080917aca8dd75c P a05f7e63e0cf44b299a3280e8360d34c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3cc8386158ac40d38bca578791f1770d in term 1.
I20260812 02:32:08.895581 19498 leader_election.cc:302] T a01e420de0e3438e9080917aca8dd75c P 3cc8386158ac40d38bca578791f1770d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3cc8386158ac40d38bca578791f1770d, b84d4904c86d4111b8ca63a0c0fe3d59; no voters:
I20260812 02:32:08.895831 19800 raft_consensus.cc:2781] T a01e420de0e3438e9080917aca8dd75c P 3cc8386158ac40d38bca578791f1770d [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:32:08.896066 19800 raft_consensus.cc:686] T a01e420de0e3438e9080917aca8dd75c P 3cc8386158ac40d38bca578791f1770d [term 1 LEADER]: Becoming Leader. State: Replica: 3cc8386158ac40d38bca578791f1770d, State: Running, Role: LEADER
I20260812 02:32:08.896473 19800 consensus_queue.cc:237] T a01e420de0e3438e9080917aca8dd75c P 3cc8386158ac40d38bca578791f1770d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a05f7e63e0cf44b299a3280e8360d34c" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 33127 } } peers { permanent_uuid: "b84d4904c86d4111b8ca63a0c0fe3d59" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 33165 } } peers { permanent_uuid: "3cc8386158ac40d38bca578791f1770d" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 36133 } }
I20260812 02:32:08.900104 19268 catalog_manager.cc:5495] T a01e420de0e3438e9080917aca8dd75c P 3cc8386158ac40d38bca578791f1770d reported cstate change: term changed from 0 to 1, leader changed from <none> to 3cc8386158ac40d38bca578791f1770d (127.14.103.2). New cstate: current_term: 1 leader_uuid: "3cc8386158ac40d38bca578791f1770d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a05f7e63e0cf44b299a3280e8360d34c" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 33127 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b84d4904c86d4111b8ca63a0c0fe3d59" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 33165 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3cc8386158ac40d38bca578791f1770d" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 36133 } health_report { overall_health: HEALTHY } } }
I20260812 02:32:08.947319 19792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:08.941560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44208 (local address 127.14.103.2:36133)
0812 02:32:08.941736 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:08.941742 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:08.941759 (+ 17us) server_negotiation.cc:407] Connection header received
0812 02:32:08.941996 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:08.941999 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:08.942060 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:08.942281 (+ 221us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 02:32:08.942614 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:08.943578 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:08.944858 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:08.945228 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:08.945341 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:08.945962 (+ 621us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 02:32:08.946671 (+ 709us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 02:32:08.946808 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0812 02:32:08.947093 (+ 285us) server_negotiation.cc:299] Negotiation successful
0812 02:32:08.947185 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260812 02:32:08.954371 19722 raft_consensus.cc:1261] T a01e420de0e3438e9080917aca8dd75c P b84d4904c86d4111b8ca63a0c0fe3d59 [term 1 FOLLOWER]: Refusing update from remote peer 3cc8386158ac40d38bca578791f1770d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:32:08.954406 19423 raft_consensus.cc:1261] T a01e420de0e3438e9080917aca8dd75c P a05f7e63e0cf44b299a3280e8360d34c [term 1 FOLLOWER]: Refusing update from remote peer 3cc8386158ac40d38bca578791f1770d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:32:08.955315 19805 consensus_queue.cc:1035] T a01e420de0e3438e9080917aca8dd75c P 3cc8386158ac40d38bca578791f1770d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b84d4904c86d4111b8ca63a0c0fe3d59" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 33165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:32:08.955545 19800 consensus_queue.cc:1035] T a01e420de0e3438e9080917aca8dd75c P 3cc8386158ac40d38bca578791f1770d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a05f7e63e0cf44b299a3280e8360d34c" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 33127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:32:08.968103 19808 mvcc.cc:205] Tried to move back new op lower bound from 7317511900988669952 to 7317511900762947584. Current Snapshot: MvccSnapshot[applied={T|T < 7317511900988669952}]
I20260812 02:32:08.978121 19267 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39008:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 02:32:08.978353 19267 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39008:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 02:32:08.982303 19267 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f696c7d60e1f48c1bbff12b98714e5cc: Sending DeleteTablet for 3 replicas of tablet a01e420de0e3438e9080917aca8dd75c
I20260812 02:32:08.983770 19403 tablet_service.cc:1514] Processing DeleteTablet for tablet a01e420de0e3438e9080917aca8dd75c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:32:08 UTC) from {username='kudu', principal='kudu/127.14.103.62@KRBTEST.COM'} at 127.0.0.1:42638
I20260812 02:32:08.983937 19697 tablet_service.cc:1514] Processing DeleteTablet for tablet a01e420de0e3438e9080917aca8dd75c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:32:08 UTC) from {username='kudu', principal='kudu/127.14.103.62@KRBTEST.COM'} at 127.0.0.1:44932
I20260812 02:32:08.984053 19548 tablet_service.cc:1514] Processing DeleteTablet for tablet a01e420de0e3438e9080917aca8dd75c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:32:08 UTC) from {username='kudu', principal='kudu/127.14.103.62@KRBTEST.COM'} at 127.0.0.1:44194
I20260812 02:32:08.984818 19823 tablet_replica.cc:330] T a01e420de0e3438e9080917aca8dd75c P 3cc8386158ac40d38bca578791f1770d: stopping tablet replica
I20260812 02:32:08.985498 19823 raft_consensus.cc:2229] T a01e420de0e3438e9080917aca8dd75c P 3cc8386158ac40d38bca578791f1770d [term 1 LEADER]: Raft consensus shutting down.
I20260812 02:32:08.985646 19821 tablet_replica.cc:330] T a01e420de0e3438e9080917aca8dd75c P b84d4904c86d4111b8ca63a0c0fe3d59: stopping tablet replica
I20260812 02:32:08.986130 19823 raft_consensus.cc:2258] T a01e420de0e3438e9080917aca8dd75c P 3cc8386158ac40d38bca578791f1770d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:32:08.986234 19821 raft_consensus.cc:2229] T a01e420de0e3438e9080917aca8dd75c P b84d4904c86d4111b8ca63a0c0fe3d59 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 02:32:08.986334 14748 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbep4M4/build/debug/bin/kudu with pid 19337
I20260812 02:32:08.988814 19821 raft_consensus.cc:2258] T a01e420de0e3438e9080917aca8dd75c P b84d4904c86d4111b8ca63a0c0fe3d59 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:32:08.991165 19823 ts_tablet_manager.cc:1905] T a01e420de0e3438e9080917aca8dd75c P 3cc8386158ac40d38bca578791f1770d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:32:08.993834 19821 ts_tablet_manager.cc:1905] T a01e420de0e3438e9080917aca8dd75c P b84d4904c86d4111b8ca63a0c0fe3d59: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:32:08.996233 19823 ts_tablet_manager.cc:1918] T a01e420de0e3438e9080917aca8dd75c P 3cc8386158ac40d38bca578791f1770d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 02:32:08.996455 19823 log.cc:1196] T a01e420de0e3438e9080917aca8dd75c P 3cc8386158ac40d38bca578791f1770d: Deleting WAL directory at /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501898713699-14748-0/minicluster-data/ts-1/wal/wals/a01e420de0e3438e9080917aca8dd75c
I20260812 02:32:08.996910 19823 ts_tablet_manager.cc:1939] T a01e420de0e3438e9080917aca8dd75c P 3cc8386158ac40d38bca578791f1770d: Deleting consensus metadata
I20260812 02:32:08.997987 19821 ts_tablet_manager.cc:1918] T a01e420de0e3438e9080917aca8dd75c P b84d4904c86d4111b8ca63a0c0fe3d59: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 02:32:08.998164 19821 log.cc:1196] T a01e420de0e3438e9080917aca8dd75c P b84d4904c86d4111b8ca63a0c0fe3d59: Deleting WAL directory at /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786501898713699-14748-0/minicluster-data/ts-2/wal/wals/a01e420de0e3438e9080917aca8dd75c
I20260812 02:32:08.998538 19821 ts_tablet_manager.cc:1939] T a01e420de0e3438e9080917aca8dd75c P b84d4904c86d4111b8ca63a0c0fe3d59: Deleting consensus metadata
I20260812 02:32:08.998723 19251 catalog_manager.cc:4841] TS 3cc8386158ac40d38bca578791f1770d (127.14.103.2:36133): tablet a01e420de0e3438e9080917aca8dd75c (table test-table [id=07347f491c784960bae27977b0b08390]) successfully deleted
I20260812 02:32:09.000723 19251 catalog_manager.cc:4841] TS b84d4904c86d4111b8ca63a0c0fe3d59 (127.14.103.3:33165): tablet a01e420de0e3438e9080917aca8dd75c (table test-table [id=07347f491c784960bae27977b0b08390]) successfully deleted
W20260812 02:32:09.002630 19254 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.103.1:33127 (error 108)
I20260812 02:32:09.003520 14748 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbep4M4/build/debug/bin/kudu with pid 19481
I20260812 02:32:09.004428 19791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:09.003734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.1:33127 (local address 127.0.0.1:42650)
0812 02:32:09.004242 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:09.004330 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.1:33127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":404}
W20260812 02:32:09.009511 19254 catalog_manager.cc:4584] TS a05f7e63e0cf44b299a3280e8360d34c (127.14.103.1:33127): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a01e420de0e3438e9080917aca8dd75c: Network error: Client connection negotiation failed: client connection to 127.14.103.1:33127: connect: Connection refused (error 111)
I20260812 02:32:09.014920 14748 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbep4M4/build/debug/bin/kudu with pid 19629
I20260812 02:32:09.025911 14748 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbep4M4/build/debug/bin/kudu with pid 19234
2026-08-12T02:32:09Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (3040 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-08-12T02:32:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T02:32:09Z Disabled control of system clock
I20260812 02:32:09.069689 14748 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbep4M4/build/debug/bin/kudu
/tmp/dist-test-taskbep4M4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786501898713699-14748-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786501898713699-14748-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786501898713699-14748-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786501898713699-14748-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.103.62:40947
--webserver_interface=127.14.103.62
--webserver_port=0
--builtin_ntp_servers=127.14.103.20:43143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.103.62:40947
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260812 02:32:09.200738 19829 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:32:09.201110 19829 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:32:09.201153 19829 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:32:09.205955 19829 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:32:09.206061 19829 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:32:09.206079 19829 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:32:09.206096 19829 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:32:09.206112 19829 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 02:32:09.211587 19829 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.103.20:43143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786501898713699-14748-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786501898713699-14748-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.103.62:40947
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.103.62:40947
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786501898713699-14748-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.103.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19829
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786501898713699-14748-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:32:09.212776 19829 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:32:09.214180 19829 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:32:09.224676 19840 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:32:09.224771 19836 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:32:09.224786 19835 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:32:09.224977 19829 server_base.cc:1034] running on GCE node
I20260812 02:32:09.226539 19829 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:32:09.228283 19829 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:32:09.229548 19829 hybrid_clock.cc:648] HybridClock initialized: now 1786501929229525 us; error 60 us; skew 500 ppm
I20260812 02:32:09.232975 19829 webserver.cc:460] Webserver started at http://127.14.103.62:39817/ using document root <none> and password file <none>
I20260812 02:32:09.233728 19829 fs_manager.cc:359] Metadata directory not provided
I20260812 02:32:09.233843 19829 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:32:09.234120 19829 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:32:09.236610 19829 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786501898713699-14748-0/minicluster-data/master-0/data/instance:
uuid: "f2404628f64b4ea897857f98548ce956"
format_stamp: "Formatted at 2026-08-12 02:32:09 on dist-test-slave-2w3w"
server_key: "1ff5a5a0029dcd417711a061b2f34d8e"
server_key_iv: "e647f593a438ea79cf4508dcdfded419"
server_key_version: "encryptionkey@0"
I20260812 02:32:09.237283 19829 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786501898713699-14748-0/minicluster-data/master-0/wal/instance:
uuid: "f2404628f64b4ea897857f98548ce956"
format_stamp: "Formatted at 2026-08-12 02:32:09 on dist-test-slave-2w3w"
server_key: "1ff5a5a0029dcd417711a061b2f34d8e"
server_key_iv: "e647f593a438ea79cf4508dcdfded419"
server_key_version: "encryptionkey@0"
I20260812 02:32:09.241937 19829 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260812 02:32:09.245669 19846 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:32:09.247668 19829 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260812 02:32:09.247839 19829 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786501898713699-14748-0/minicluster-data/master-0/data,/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786501898713699-14748-0/minicluster-data/master-0/wal
uuid: "f2404628f64b4ea897857f98548ce956"
format_stamp: "Formatted at 2026-08-12 02:32:09 on dist-test-slave-2w3w"
server_key: "1ff5a5a0029dcd417711a061b2f34d8e"
server_key_iv: "e647f593a438ea79cf4508dcdfded419"
server_key_version: "encryptionkey@0"
I20260812 02:32:09.248049 19829 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786501898713699-14748-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786501898713699-14748-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786501898713699-14748-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:32:09.253381 19829 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:32:09.254172 19829 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:32:09.254388 19829 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:32:09.262662 19829 rpc_server.cc:307] RPC server started. Bound to: 127.14.103.62:40947
I20260812 02:32:09.262784 19906 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.103.62:40947 every 8 connection(s)
I20260812 02:32:09.263825 19829 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786501898713699-14748-0/minicluster-data/master-0/data/info.pb
I20260812 02:32:09.268057 19907 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:32:09.269291 14748 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbep4M4/build/debug/bin/kudu as pid 19829
I20260812 02:32:09.269435 14748 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786501898713699-14748-0/minicluster-data/master-0/wal/instance
I20260812 02:32:09.269778 14748 external_mini_cluster.cc:1428] Setting key 35df8f8a28b7e76b5d3b8a4b98d967a4
I20260812 02:32:09.275141 19907 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2404628f64b4ea897857f98548ce956: Bootstrap starting.
I20260812 02:32:09.276065 19910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:09.271461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40010 (local address 127.14.103.62:40947)
0812 02:32:09.272240 (+ 779us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:09.272258 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:09.272297 (+ 39us) server_negotiation.cc:407] Connection header received
0812 02:32:09.273006 (+ 709us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:09.273025 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:09.273412 (+ 387us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:09.273778 (+ 366us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:32:09.274640 (+ 862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:32:09.274645 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:32:09.274658 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:32:09.274670 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:32:09.274729 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:32:09.274759 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:32:09.274929 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0812 02:32:09.275031 (+ 102us) server_negotiation.cc:299] Negotiation successful
0812 02:32:09.275322 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374,"thread_start_us":153,"threads_started":1}
I20260812 02:32:09.277670 19907 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f2404628f64b4ea897857f98548ce956: Neither blocks nor log segments found. Creating new log.
I20260812 02:32:09.278635 19907 log.cc:824] T 00000000000000000000000000000000 P f2404628f64b4ea897857f98548ce956: Log is configured to *not* fsync() on all Append() calls
I20260812 02:32:09.281422 19907 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2404628f64b4ea897857f98548ce956: No bootstrap required, opened a new log
I20260812 02:32:09.287195 19907 raft_consensus.cc:348] T 00000000000000000000000000000000 P f2404628f64b4ea897857f98548ce956 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2404628f64b4ea897857f98548ce956" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 40947 } }
I20260812 02:32:09.287499 19907 raft_consensus.cc:374] T 00000000000000000000000000000000 P f2404628f64b4ea897857f98548ce956 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:32:09.287535 19907 raft_consensus.cc:729] T 00000000000000000000000000000000 P f2404628f64b4ea897857f98548ce956 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2404628f64b4ea897857f98548ce956, State: Initialized, Role: FOLLOWER
I20260812 02:32:09.288071 19907 consensus_queue.cc:260] T 00000000000000000000000000000000 P f2404628f64b4ea897857f98548ce956 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2404628f64b4ea897857f98548ce956" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 40947 } }
I20260812 02:32:09.288221 19907 raft_consensus.cc:388] T 00000000000000000000000000000000 P f2404628f64b4ea897857f98548ce956 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 02:32:09.288270 19907 raft_consensus.cc:482] T 00000000000000000000000000000000 P f2404628f64b4ea897857f98548ce956 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 02:32:09.288344 19907 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f2404628f64b4ea897857f98548ce956 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:32:09.289572 19907 raft_consensus.cc:504] T 00000000000000000000000000000000 P f2404628f64b4ea897857f98548ce956 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2404628f64b4ea897857f98548ce956" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 40947 } }
I20260812 02:32:09.289992 19907 leader_election.cc:302] T 00000000000000000000000000000000 P f2404628f64b4ea897857f98548ce956 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f2404628f64b4ea897857f98548ce956; no voters:
I20260812 02:32:09.290676 19907 leader_election.cc:288] T 00000000000000000000000000000000 P f2404628f64b4ea897857f98548ce956 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 02:32:09.290774 19912 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f2404628f64b4ea897857f98548ce956 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:32:09.291141 19912 raft_consensus.cc:686] T 00000000000000000000000000000000 P f2404628f64b4ea897857f98548ce956 [term 1 LEADER]: Becoming Leader. State: Replica: f2404628f64b4ea897857f98548ce956, State: Running, Role: LEADER
I20260812 02:32:09.291565 19912 consensus_queue.cc:237] T 00000000000000000000000000000000 P f2404628f64b4ea897857f98548ce956 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2404628f64b4ea897857f98548ce956" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 40947 } }
I20260812 02:32:09.292179 19907 sys_catalog.cc:564] T 00000000000000000000000000000000 P f2404628f64b4ea897857f98548ce956 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:32:09.293362 19914 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2404628f64b4ea897857f98548ce956 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f2404628f64b4ea897857f98548ce956. Latest consensus state: current_term: 1 leader_uuid: "f2404628f64b4ea897857f98548ce956" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2404628f64b4ea897857f98548ce956" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 40947 } } }
I20260812 02:32:09.293515 19914 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2404628f64b4ea897857f98548ce956 [sys.catalog]: This master's current role is: LEADER
I20260812 02:32:09.293784 19913 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2404628f64b4ea897857f98548ce956 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f2404628f64b4ea897857f98548ce956" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2404628f64b4ea897857f98548ce956" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 40947 } } }
I20260812 02:32:09.293917 19921 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 02:32:09.294056 19913 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2404628f64b4ea897857f98548ce956 [sys.catalog]: This master's current role is: LEADER
I20260812 02:32:09.297691 19921 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 02:32:09.305004 19921 catalog_manager.cc:1347] Generated new cluster ID: 67251013002d4f23874942883397dd25
I20260812 02:32:09.305112 19921 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 02:32:09.321635 19921 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 02:32:09.322052 19921 catalog_manager.cc:1504] Loading token signing keys...
I20260812 02:32:09.329505 19921 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f2404628f64b4ea897857f98548ce956: Generated new TSK 0
I20260812 02:32:09.330351 19921 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 02:32:09.334790 14748 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbep4M4/build/debug/bin/kudu
/tmp/dist-test-taskbep4M4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786501898713699-14748-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786501898713699-14748-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786501898713699-14748-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786501898713699-14748-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.103.1:0
--local_ip_for_outbound_sockets=127.14.103.1
--webserver_interface=127.14.103.1
--webserver_port=0
--tserver_master_addrs=127.14.103.62:40947
--builtin_ntp_servers=127.14.103.20:43143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260812 02:32:09.481490 19934 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:32:09.481828 19934 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:32:09.481875 19934 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:32:09.486351 19934 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:32:09.486446 19934 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:32:09.486512 19934 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.103.1
I20260812 02:32:09.491860 19934 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.103.20:43143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786501898713699-14748-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786501898713699-14748-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.103.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786501898713699-14748-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.103.1
--webserver_port=0
--tserver_master_addrs=127.14.103.62:40947
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19934
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.103.1
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786501898713699-14748-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:32:09.493233 19934 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:32:09.495049 19934 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:32:09.507906 19940 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:32:09.507906 19943 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:32:09.508397 19941 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:32:09.508468 19934 server_base.cc:1034] running on GCE node
I20260812 02:32:09.509014 19934 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:32:09.509912 19934 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:32:09.511312 19934 hybrid_clock.cc:648] HybridClock initialized: now 1786501929511218 us; error 101 us; skew 500 ppm
I20260812 02:32:09.513844 19934 webserver.cc:460] Webserver started at http://127.14.103.1:41575/ using document root <none> and password file <none>
I20260812 02:32:09.514566 19934 fs_manager.cc:359] Metadata directory not provided
I20260812 02:32:09.514680 19934 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:32:09.515046 19934 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:32:09.517572 19934 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786501898713699-14748-0/minicluster-data/ts-0/data/instance:
uuid: "5412036532af43dd9fec36a0aa497a41"
format_stamp: "Formatted at 2026-08-12 02:32:09 on dist-test-slave-2w3w"
server_key: "b2a873043925b848297dc0b70c9dc432"
server_key_iv: "cfb90b12889f0515b2960af04bdddf8e"
server_key_version: "encryptionkey@0"
I20260812 02:32:09.518321 19934 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786501898713699-14748-0/minicluster-data/ts-0/wal/instance:
uuid: "5412036532af43dd9fec36a0aa497a41"
format_stamp: "Formatted at 2026-08-12 02:32:09 on dist-test-slave-2w3w"
server_key: "b2a873043925b848297dc0b70c9dc432"
server_key_iv: "cfb90b12889f0515b2960af04bdddf8e"
server_key_version: "encryptionkey@0"
I20260812 02:32:09.522922 19934 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260812 02:32:09.526134 19951 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:32:09.527786 19934 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 02:32:09.527983 19934 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786501898713699-14748-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786501898713699-14748-0/minicluster-data/ts-0/wal
uuid: "5412036532af43dd9fec36a0aa497a41"
format_stamp: "Formatted at 2026-08-12 02:32:09 on dist-test-slave-2w3w"
server_key: "b2a873043925b848297dc0b70c9dc432"
server_key_iv: "cfb90b12889f0515b2960af04bdddf8e"
server_key_version: "encryptionkey@0"
I20260812 02:32:09.528146 19934 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786501898713699-14748-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786501898713699-14748-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786501898713699-14748-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:32:09.534087 19934 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:32:09.534945 19934 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:32:09.535244 19934 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:32:09.535938 19934 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:32:09.537387 19934 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:32:09.537492 19934 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:32:09.537572 19934 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:32:09.537626 19934 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:32:09.549460 19934 rpc_server.cc:307] RPC server started. Bound to: 127.14.103.1:44115
I20260812 02:32:09.549505 20076 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.103.1:44115 every 8 connection(s)
I20260812 02:32:09.550558 19934 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786501898713699-14748-0/minicluster-data/ts-0/data/info.pb
I20260812 02:32:09.554839 14748 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbep4M4/build/debug/bin/kudu as pid 19934
I20260812 02:32:09.555011 14748 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786501898713699-14748-0/minicluster-data/ts-0/wal/instance
I20260812 02:32:09.555377 14748 external_mini_cluster.cc:1428] Setting key 9882592e130f92620357ea9d26b7ee18
I20260812 02:32:09.558141 14748 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbep4M4/build/debug/bin/kudu
/tmp/dist-test-taskbep4M4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786501898713699-14748-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786501898713699-14748-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786501898713699-14748-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786501898713699-14748-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.103.2:0
--local_ip_for_outbound_sockets=127.14.103.2
--webserver_interface=127.14.103.2
--webserver_port=0
--tserver_master_addrs=127.14.103.62:40947
--builtin_ntp_servers=127.14.103.20:43143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260812 02:32:09.559260 19910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:09.553127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.1:37649 (local address 127.14.103.62:40947)
0812 02:32:09.553262 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:09.553266 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:09.554847 (+ 1581us) server_negotiation.cc:407] Connection header received
0812 02:32:09.556831 (+ 1984us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:09.556835 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:09.556881 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:09.557040 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:32:09.558540 (+ 1500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:32:09.558544 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:32:09.558546 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:32:09.558566 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:32:09.558588 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:32:09.558597 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:32:09.558694 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0812 02:32:09.559086 (+ 392us) server_negotiation.cc:299] Negotiation successful
0812 02:32:09.559138 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260812 02:32:09.560783 20079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:09.553620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:40947 (local address 127.14.103.1:37649)
0812 02:32:09.554589 (+ 969us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:09.554655 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0812 02:32:09.556560 (+ 1905us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:32:09.557261 (+ 701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:32:09.557270 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:32:09.557416 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:32:09.558117 (+ 701us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 02:32:09.558119 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:32:09.558185 (+ 66us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 02:32:09.558296 (+ 111us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:32:09.558741 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:32:09.558756 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:32:09.558767 (+ 11us) client_negotiation.cc:769] Sending connection context
0812 02:32:09.559052 (+ 285us) client_negotiation.cc:240] Negotiation successful
0812 02:32:09.559582 (+ 530us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":125,"threads_started":1}
I20260812 02:32:09.562618 20077 heartbeater.cc:347] Connected to a master server at 127.14.103.62:40947
I20260812 02:32:09.562940 20077 heartbeater.cc:464] Registering TS with master...
I20260812 02:32:09.563524 20077 heartbeater.cc:511] Master 127.14.103.62:40947 requested a full tablet report, sending...
I20260812 02:32:09.566716 19868 ts_manager.cc:194] Registered new tserver with Master: 5412036532af43dd9fec36a0aa497a41 (127.14.103.1:44115)
W20260812 02:32:09.700889 20080 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:32:09.701385 20080 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:32:09.701449 20080 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:32:09.706409 20080 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:32:09.706524 20080 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:32:09.706591 20080 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.103.2
I20260812 02:32:09.712597 20080 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.103.20:43143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786501898713699-14748-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786501898713699-14748-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.103.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786501898713699-14748-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.103.2
--webserver_port=0
--tserver_master_addrs=127.14.103.62:40947
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20080
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.103.2
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786501898713699-14748-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:32:09.714100 20080 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:32:09.715883 20080 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:32:09.727880 20086 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:32:09.728152 20088 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:32:09.727856 20085 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:32:09.729192 20080 server_base.cc:1034] running on GCE node
I20260812 02:32:09.729790 20080 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:32:09.730587 20080 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:32:09.731804 20080 hybrid_clock.cc:648] HybridClock initialized: now 1786501929731758 us; error 71 us; skew 500 ppm
I20260812 02:32:09.735131 20080 webserver.cc:460] Webserver started at http://127.14.103.2:45205/ using document root <none> and password file <none>
I20260812 02:32:09.735904 20080 fs_manager.cc:359] Metadata directory not provided
I20260812 02:32:09.736024 20080 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:32:09.736294 20080 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:32:09.738713 20080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786501898713699-14748-0/minicluster-data/ts-1/data/instance:
uuid: "fe6b7b5d2781471da5124683b456b7bd"
format_stamp: "Formatted at 2026-08-12 02:32:09 on dist-test-slave-2w3w"
server_key: "b323acd5a82326d93b289d97e7331064"
server_key_iv: "519b3c1dbb260e843dd726dfe3b268a4"
server_key_version: "encryptionkey@0"
I20260812 02:32:09.739835 20080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786501898713699-14748-0/minicluster-data/ts-1/wal/instance:
uuid: "fe6b7b5d2781471da5124683b456b7bd"
format_stamp: "Formatted at 2026-08-12 02:32:09 on dist-test-slave-2w3w"
server_key: "b323acd5a82326d93b289d97e7331064"
server_key_iv: "519b3c1dbb260e843dd726dfe3b268a4"
server_key_version: "encryptionkey@0"
I20260812 02:32:09.744901 20080 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260812 02:32:09.748826 20094 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:32:09.750388 20080 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260812 02:32:09.750569 20080 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786501898713699-14748-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786501898713699-14748-0/minicluster-data/ts-1/wal
uuid: "fe6b7b5d2781471da5124683b456b7bd"
format_stamp: "Formatted at 2026-08-12 02:32:09 on dist-test-slave-2w3w"
server_key: "b323acd5a82326d93b289d97e7331064"
server_key_iv: "519b3c1dbb260e843dd726dfe3b268a4"
server_key_version: "encryptionkey@0"
I20260812 02:32:09.750700 20080 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786501898713699-14748-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786501898713699-14748-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786501898713699-14748-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:32:09.756829 20080 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:32:09.757619 20080 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:32:09.757849 20080 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:32:09.758671 20080 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:32:09.759910 20080 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:32:09.759995 20080 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:32:09.760076 20080 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:32:09.760120 20080 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:32:09.771198 20080 rpc_server.cc:307] RPC server started. Bound to: 127.14.103.2:41665
I20260812 02:32:09.771221 20222 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.103.2:41665 every 8 connection(s)
I20260812 02:32:09.772758 20080 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786501898713699-14748-0/minicluster-data/ts-1/data/info.pb
I20260812 02:32:09.779592 14748 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbep4M4/build/debug/bin/kudu as pid 20080
I20260812 02:32:09.779803 14748 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786501898713699-14748-0/minicluster-data/ts-1/wal/instance
I20260812 02:32:09.780412 14748 external_mini_cluster.cc:1428] Setting key 990986ff82090cf31102b7bdcd193a4e
I20260812 02:32:09.782038 19910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:09.776014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.2:38385 (local address 127.14.103.62:40947)
0812 02:32:09.776218 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:09.776224 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:09.777564 (+ 1340us) server_negotiation.cc:407] Connection header received
0812 02:32:09.779155 (+ 1591us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:09.779161 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:09.779243 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:09.779403 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:32:09.781424 (+ 2021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:32:09.781429 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:32:09.781431 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:32:09.781448 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:32:09.781511 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:32:09.781522 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:32:09.781652 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0812 02:32:09.781853 (+ 201us) server_negotiation.cc:299] Negotiation successful
0812 02:32:09.781902 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260812 02:32:09.783623 14748 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbep4M4/build/debug/bin/kudu
/tmp/dist-test-taskbep4M4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786501898713699-14748-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786501898713699-14748-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786501898713699-14748-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786501898713699-14748-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.103.3:0
--local_ip_for_outbound_sockets=127.14.103.3
--webserver_interface=127.14.103.3
--webserver_port=0
--tserver_master_addrs=127.14.103.62:40947
--builtin_ntp_servers=127.14.103.20:43143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260812 02:32:09.783846 20225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:09.776565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:40947 (local address 127.14.103.2:38385)
0812 02:32:09.777366 (+ 801us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:09.777413 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0812 02:32:09.778722 (+ 1309us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:32:09.779794 (+ 1072us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:32:09.779806 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:32:09.779957 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:32:09.780939 (+ 982us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 02:32:09.780941 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:32:09.781042 (+ 101us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 02:32:09.781187 (+ 145us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:32:09.781670 (+ 483us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:32:09.781679 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:32:09.781685 (+ 6us) client_negotiation.cc:769] Sending connection context
0812 02:32:09.781878 (+ 193us) client_negotiation.cc:240] Negotiation successful
0812 02:32:09.782436 (+ 558us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":161,"threads_started":1}
I20260812 02:32:09.785552 20223 heartbeater.cc:347] Connected to a master server at 127.14.103.62:40947
I20260812 02:32:09.785912 20223 heartbeater.cc:464] Registering TS with master...
I20260812 02:32:09.786859 20223 heartbeater.cc:511] Master 127.14.103.62:40947 requested a full tablet report, sending...
I20260812 02:32:09.788730 19868 ts_manager.cc:194] Registered new tserver with Master: fe6b7b5d2781471da5124683b456b7bd (127.14.103.2:41665)
W20260812 02:32:09.918381 20226 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:32:09.918740 20226 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:32:09.918825 20226 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:32:09.923491 20226 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:32:09.923588 20226 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:32:09.923671 20226 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.103.3
I20260812 02:32:09.929512 20226 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.103.20:43143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786501898713699-14748-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786501898713699-14748-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.103.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786501898713699-14748-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.103.3
--webserver_port=0
--tserver_master_addrs=127.14.103.62:40947
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20226
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.103.3
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786501898713699-14748-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:32:09.930661 20226 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:32:09.932364 20226 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:32:09.943338 20232 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:32:09.943519 20226 server_base.cc:1034] running on GCE node
W20260812 02:32:09.943342 20234 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:32:09.943785 20231 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:32:09.944502 20226 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:32:09.945463 20226 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:32:09.946677 20226 hybrid_clock.cc:648] HybridClock initialized: now 1786501929946662 us; error 39 us; skew 500 ppm
I20260812 02:32:09.954641 20226 webserver.cc:460] Webserver started at http://127.14.103.3:38165/ using document root <none> and password file <none>
I20260812 02:32:09.955528 20226 fs_manager.cc:359] Metadata directory not provided
I20260812 02:32:09.955610 20226 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:32:09.955862 20226 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:32:09.958663 20226 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786501898713699-14748-0/minicluster-data/ts-2/data/instance:
uuid: "1d1aad8de5434b16858d1c54bb71922f"
format_stamp: "Formatted at 2026-08-12 02:32:09 on dist-test-slave-2w3w"
server_key: "f9d793ad3ece96453c6301f7346b9848"
server_key_iv: "290d6004ccea39ef3cf489dba8d017ee"
server_key_version: "encryptionkey@0"
I20260812 02:32:09.959705 20226 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786501898713699-14748-0/minicluster-data/ts-2/wal/instance:
uuid: "1d1aad8de5434b16858d1c54bb71922f"
format_stamp: "Formatted at 2026-08-12 02:32:09 on dist-test-slave-2w3w"
server_key: "f9d793ad3ece96453c6301f7346b9848"
server_key_iv: "290d6004ccea39ef3cf489dba8d017ee"
server_key_version: "encryptionkey@0"
I20260812 02:32:09.964663 20226 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260812 02:32:09.968109 20240 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:32:09.969719 20226 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 02:32:09.969933 20226 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786501898713699-14748-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786501898713699-14748-0/minicluster-data/ts-2/wal
uuid: "1d1aad8de5434b16858d1c54bb71922f"
format_stamp: "Formatted at 2026-08-12 02:32:09 on dist-test-slave-2w3w"
server_key: "f9d793ad3ece96453c6301f7346b9848"
server_key_iv: "290d6004ccea39ef3cf489dba8d017ee"
server_key_version: "encryptionkey@0"
I20260812 02:32:09.970096 20226 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786501898713699-14748-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786501898713699-14748-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786501898713699-14748-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:32:09.975394 20226 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:32:09.976120 20226 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:32:09.976333 20226 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:32:09.976948 20226 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:32:09.977972 20226 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:32:09.978046 20226 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:32:09.978122 20226 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:32:09.978165 20226 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:32:09.989038 20226 rpc_server.cc:307] RPC server started. Bound to: 127.14.103.3:34049
I20260812 02:32:09.989104 20364 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.103.3:34049 every 8 connection(s)
I20260812 02:32:09.990160 20226 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786501898713699-14748-0/minicluster-data/ts-2/data/info.pb
I20260812 02:32:09.993896 14748 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbep4M4/build/debug/bin/kudu as pid 20226
I20260812 02:32:09.994038 14748 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786501898713699-14748-0/minicluster-data/ts-2/wal/instance
I20260812 02:32:09.994390 14748 external_mini_cluster.cc:1428] Setting key d3fdb98714e4bc6f16492bdd1e41b262
I20260812 02:32:09.997805 19910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:09.992581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.3:43143 (local address 127.14.103.62:40947)
0812 02:32:09.992822 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:09.992826 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:09.994309 (+ 1483us) server_negotiation.cc:407] Connection header received
0812 02:32:09.995973 (+ 1664us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:09.995977 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:09.996029 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:09.996107 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:32:09.997367 (+ 1260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:32:09.997371 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:32:09.997373 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:32:09.997386 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:32:09.997407 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:32:09.997415 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:32:09.997502 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0812 02:32:09.997644 (+ 142us) server_negotiation.cc:299] Negotiation successful
0812 02:32:09.997685 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260812 02:32:09.998812 20367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:09.993064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:40947 (local address 127.14.103.3:43143)
0812 02:32:09.994063 (+ 999us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:09.994133 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0812 02:32:09.995712 (+ 1579us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:32:09.996304 (+ 592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:32:09.996312 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:32:09.996417 (+ 105us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:32:09.997044 (+ 627us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 02:32:09.997045 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:32:09.997092 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 02:32:09.997168 (+ 76us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:32:09.997505 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:32:09.997512 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:32:09.997517 (+ 5us) client_negotiation.cc:769] Sending connection context
0812 02:32:09.997668 (+ 151us) client_negotiation.cc:240] Negotiation successful
0812 02:32:09.997886 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":176,"threads_started":1}
I20260812 02:32:10.000244 20365 heartbeater.cc:347] Connected to a master server at 127.14.103.62:40947
I20260812 02:32:10.000561 20365 heartbeater.cc:464] Registering TS with master...
I20260812 02:32:10.001077 20365 heartbeater.cc:511] Master 127.14.103.62:40947 requested a full tablet report, sending...
I20260812 02:32:10.002409 19868 ts_manager.cc:194] Registered new tserver with Master: 1d1aad8de5434b16858d1c54bb71922f (127.14.103.3:34049)
I20260812 02:32:10.009712 14748 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 02:32:10.014082 19910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:10.012092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40024 (local address 127.14.103.62:40947)
0812 02:32:10.012258 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:10.012262 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:10.012389 (+ 127us) server_negotiation.cc:407] Connection header received
0812 02:32:10.012496 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:10.012498 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:10.012535 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:10.012610 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:32:10.013644 (+ 1034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:32:10.013647 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:32:10.013649 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:32:10.013661 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:32:10.013680 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:32:10.013688 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:32:10.013774 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0812 02:32:10.013931 (+ 157us) server_negotiation.cc:299] Negotiation successful
0812 02:32:10.013968 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260812 02:32:10.017742 19868 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40024:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 02:32:10.021095 19868 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 02:32:10.034979 20382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:10.031968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35802 (local address 127.14.103.3:34049)
0812 02:32:10.032701 (+ 733us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:10.032712 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:10.032739 (+ 27us) server_negotiation.cc:407] Connection header received
0812 02:32:10.032796 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:10.032805 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:10.033052 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:10.033213 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:32:10.034247 (+ 1034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:32:10.034257 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:32:10.034273 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:32:10.034289 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:32:10.034341 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:32:10.034378 (+ 37us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:32:10.034490 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0812 02:32:10.034704 (+ 214us) server_negotiation.cc:299] Negotiation successful
0812 02:32:10.034797 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":606,"thread_start_us":156,"threads_started":1}
I20260812 02:32:10.034952 20376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:10.031506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.3:34049 (local address 127.0.0.1:35802)
0812 02:32:10.032596 (+ 1090us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:10.032612 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 02:32:10.032697 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:32:10.033293 (+ 596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:32:10.033298 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:32:10.033382 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:32:10.034031 (+ 649us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 02:32:10.034032 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:32:10.034051 (+ 19us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 02:32:10.034105 (+ 54us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:32:10.034537 (+ 432us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:32:10.034548 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:32:10.034554 (+ 6us) client_negotiation.cc:769] Sending connection context
0812 02:32:10.034689 (+ 135us) client_negotiation.cc:240] Negotiation successful
0812 02:32:10.034783 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1020,"thread_start_us":117,"threads_started":1}
I20260812 02:32:10.035430 20379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:10.031267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.2:41665 (local address 127.0.0.1:52508)
0812 02:32:10.032121 (+ 854us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:10.032179 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0812 02:32:10.032279 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:32:10.034134 (+ 1855us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:32:10.034137 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:32:10.034150 (+ 13us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:32:10.034666 (+ 516us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 02:32:10.034668 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:32:10.034677 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 02:32:10.034699 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:32:10.035163 (+ 464us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:32:10.035166 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:32:10.035169 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 02:32:10.035246 (+ 77us) client_negotiation.cc:240] Negotiation successful
0812 02:32:10.035283 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":672,"thread_start_us":113,"threads_started":1}
I20260812 02:32:10.036062 20380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:10.031860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33596 (local address 127.14.103.1:44115)
0812 02:32:10.032257 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:10.032452 (+ 195us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:10.032976 (+ 524us) server_negotiation.cc:407] Connection header received
0812 02:32:10.033040 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:10.033045 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:10.033199 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:10.033318 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:32:10.034675 (+ 1357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:32:10.034681 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:32:10.034693 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:32:10.034706 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:32:10.034765 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:32:10.034804 (+ 39us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:32:10.035288 (+ 484us) server_negotiation.cc:1035] Waiting for connection context
0812 02:32:10.035671 (+ 383us) server_negotiation.cc:299] Negotiation successful
0812 02:32:10.035904 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":89,"threads_started":1}
I20260812 02:32:10.036137 20378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:10.031436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52508 (local address 127.14.103.2:41665)
0812 02:32:10.032695 (+ 1259us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:10.032711 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:10.032738 (+ 27us) server_negotiation.cc:407] Connection header received
0812 02:32:10.032798 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:10.032806 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:10.033047 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:10.033200 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:32:10.034860 (+ 1660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:32:10.034869 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:32:10.034882 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:32:10.034898 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:32:10.034939 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:32:10.035011 (+ 72us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:32:10.035117 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0812 02:32:10.035882 (+ 765us) server_negotiation.cc:299] Negotiation successful
0812 02:32:10.035981 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1142,"thread_start_us":127,"threads_started":1}
I20260812 02:32:10.035578 20377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:10.031104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.1:44115 (local address 127.0.0.1:33596)
0812 02:32:10.032862 (+ 1758us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:10.032874 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 02:32:10.032958 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:32:10.033747 (+ 789us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:32:10.033750 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:32:10.033765 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:32:10.034473 (+ 708us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 02:32:10.034476 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:32:10.034488 (+ 12us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 02:32:10.034519 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:32:10.035331 (+ 812us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:32:10.035335 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:32:10.035338 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 02:32:10.035407 (+ 69us) client_negotiation.cc:240] Negotiation successful
0812 02:32:10.035437 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":688,"thread_start_us":90,"threads_started":1}
I20260812 02:32:10.038455 20295 tablet_service.cc:1467] Processing CreateTablet for tablet 184a63d17b104e98aaedceddf554fb34 (DEFAULT_TABLE table=test-table [id=e1f4ddc3f78a497690e089b390fd142e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:32:10.038938 20148 tablet_service.cc:1467] Processing CreateTablet for tablet 184a63d17b104e98aaedceddf554fb34 (DEFAULT_TABLE table=test-table [id=e1f4ddc3f78a497690e089b390fd142e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:32:10.039113 20005 tablet_service.cc:1467] Processing CreateTablet for tablet 184a63d17b104e98aaedceddf554fb34 (DEFAULT_TABLE table=test-table [id=e1f4ddc3f78a497690e089b390fd142e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:32:10.039780 20295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 184a63d17b104e98aaedceddf554fb34. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:32:10.040166 20005 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 184a63d17b104e98aaedceddf554fb34. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:32:10.040166 20148 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 184a63d17b104e98aaedceddf554fb34. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:32:10.047807 20386 tablet_bootstrap.cc:492] T 184a63d17b104e98aaedceddf554fb34 P 1d1aad8de5434b16858d1c54bb71922f: Bootstrap starting.
I20260812 02:32:10.048317 20384 tablet_bootstrap.cc:492] T 184a63d17b104e98aaedceddf554fb34 P fe6b7b5d2781471da5124683b456b7bd: Bootstrap starting.
I20260812 02:32:10.049132 20385 tablet_bootstrap.cc:492] T 184a63d17b104e98aaedceddf554fb34 P 5412036532af43dd9fec36a0aa497a41: Bootstrap starting.
I20260812 02:32:10.051905 20384 tablet_bootstrap.cc:654] T 184a63d17b104e98aaedceddf554fb34 P fe6b7b5d2781471da5124683b456b7bd: Neither blocks nor log segments found. Creating new log.
I20260812 02:32:10.051905 20385 tablet_bootstrap.cc:654] T 184a63d17b104e98aaedceddf554fb34 P 5412036532af43dd9fec36a0aa497a41: Neither blocks nor log segments found. Creating new log.
I20260812 02:32:10.052935 20385 log.cc:824] T 184a63d17b104e98aaedceddf554fb34 P 5412036532af43dd9fec36a0aa497a41: Log is configured to *not* fsync() on all Append() calls
I20260812 02:32:10.052935 20384 log.cc:824] T 184a63d17b104e98aaedceddf554fb34 P fe6b7b5d2781471da5124683b456b7bd: Log is configured to *not* fsync() on all Append() calls
I20260812 02:32:10.053608 20386 tablet_bootstrap.cc:654] T 184a63d17b104e98aaedceddf554fb34 P 1d1aad8de5434b16858d1c54bb71922f: Neither blocks nor log segments found. Creating new log.
I20260812 02:32:10.054526 20386 log.cc:824] T 184a63d17b104e98aaedceddf554fb34 P 1d1aad8de5434b16858d1c54bb71922f: Log is configured to *not* fsync() on all Append() calls
I20260812 02:32:10.055516 20384 tablet_bootstrap.cc:492] T 184a63d17b104e98aaedceddf554fb34 P fe6b7b5d2781471da5124683b456b7bd: No bootstrap required, opened a new log
I20260812 02:32:10.055516 20385 tablet_bootstrap.cc:492] T 184a63d17b104e98aaedceddf554fb34 P 5412036532af43dd9fec36a0aa497a41: No bootstrap required, opened a new log
I20260812 02:32:10.055737 20384 ts_tablet_manager.cc:1397] T 184a63d17b104e98aaedceddf554fb34 P fe6b7b5d2781471da5124683b456b7bd: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260812 02:32:10.055737 20385 ts_tablet_manager.cc:1397] T 184a63d17b104e98aaedceddf554fb34 P 5412036532af43dd9fec36a0aa497a41: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260812 02:32:10.061021 20386 tablet_bootstrap.cc:492] T 184a63d17b104e98aaedceddf554fb34 P 1d1aad8de5434b16858d1c54bb71922f: No bootstrap required, opened a new log
I20260812 02:32:10.061448 20386 ts_tablet_manager.cc:1397] T 184a63d17b104e98aaedceddf554fb34 P 1d1aad8de5434b16858d1c54bb71922f: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.004s
I20260812 02:32:10.061053 20384 raft_consensus.cc:348] T 184a63d17b104e98aaedceddf554fb34 P fe6b7b5d2781471da5124683b456b7bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5412036532af43dd9fec36a0aa497a41" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 44115 } } peers { permanent_uuid: "fe6b7b5d2781471da5124683b456b7bd" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 41665 } } peers { permanent_uuid: "1d1aad8de5434b16858d1c54bb71922f" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 34049 } }
I20260812 02:32:10.061609 20384 raft_consensus.cc:374] T 184a63d17b104e98aaedceddf554fb34 P fe6b7b5d2781471da5124683b456b7bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:32:10.061053 20385 raft_consensus.cc:348] T 184a63d17b104e98aaedceddf554fb34 P 5412036532af43dd9fec36a0aa497a41 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5412036532af43dd9fec36a0aa497a41" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 44115 } } peers { permanent_uuid: "fe6b7b5d2781471da5124683b456b7bd" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 41665 } } peers { permanent_uuid: "1d1aad8de5434b16858d1c54bb71922f" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 34049 } }
I20260812 02:32:10.061689 20385 raft_consensus.cc:374] T 184a63d17b104e98aaedceddf554fb34 P 5412036532af43dd9fec36a0aa497a41 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:32:10.061690 20384 raft_consensus.cc:729] T 184a63d17b104e98aaedceddf554fb34 P fe6b7b5d2781471da5124683b456b7bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe6b7b5d2781471da5124683b456b7bd, State: Initialized, Role: FOLLOWER
I20260812 02:32:10.061753 20385 raft_consensus.cc:729] T 184a63d17b104e98aaedceddf554fb34 P 5412036532af43dd9fec36a0aa497a41 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5412036532af43dd9fec36a0aa497a41, State: Initialized, Role: FOLLOWER
I20260812 02:32:10.062290 20385 consensus_queue.cc:260] T 184a63d17b104e98aaedceddf554fb34 P 5412036532af43dd9fec36a0aa497a41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5412036532af43dd9fec36a0aa497a41" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 44115 } } peers { permanent_uuid: "fe6b7b5d2781471da5124683b456b7bd" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 41665 } } peers { permanent_uuid: "1d1aad8de5434b16858d1c54bb71922f" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 34049 } }
I20260812 02:32:10.062290 20384 consensus_queue.cc:260] T 184a63d17b104e98aaedceddf554fb34 P fe6b7b5d2781471da5124683b456b7bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5412036532af43dd9fec36a0aa497a41" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 44115 } } peers { permanent_uuid: "fe6b7b5d2781471da5124683b456b7bd" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 41665 } } peers { permanent_uuid: "1d1aad8de5434b16858d1c54bb71922f" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 34049 } }
I20260812 02:32:10.063560 20385 ts_tablet_manager.cc:1428] T 184a63d17b104e98aaedceddf554fb34 P 5412036532af43dd9fec36a0aa497a41: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260812 02:32:10.063560 20384 ts_tablet_manager.cc:1428] T 184a63d17b104e98aaedceddf554fb34 P fe6b7b5d2781471da5124683b456b7bd: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260812 02:32:10.063978 20223 heartbeater.cc:503] Master 127.14.103.62:40947 was elected leader, sending a full tablet report...
I20260812 02:32:10.064199 20077 heartbeater.cc:503] Master 127.14.103.62:40947 was elected leader, sending a full tablet report...
I20260812 02:32:10.066295 20386 raft_consensus.cc:348] T 184a63d17b104e98aaedceddf554fb34 P 1d1aad8de5434b16858d1c54bb71922f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5412036532af43dd9fec36a0aa497a41" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 44115 } } peers { permanent_uuid: "fe6b7b5d2781471da5124683b456b7bd" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 41665 } } peers { permanent_uuid: "1d1aad8de5434b16858d1c54bb71922f" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 34049 } }
I20260812 02:32:10.066658 20386 raft_consensus.cc:374] T 184a63d17b104e98aaedceddf554fb34 P 1d1aad8de5434b16858d1c54bb71922f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:32:10.066747 20386 raft_consensus.cc:729] T 184a63d17b104e98aaedceddf554fb34 P 1d1aad8de5434b16858d1c54bb71922f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d1aad8de5434b16858d1c54bb71922f, State: Initialized, Role: FOLLOWER
I20260812 02:32:10.067307 20386 consensus_queue.cc:260] T 184a63d17b104e98aaedceddf554fb34 P 1d1aad8de5434b16858d1c54bb71922f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5412036532af43dd9fec36a0aa497a41" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 44115 } } peers { permanent_uuid: "fe6b7b5d2781471da5124683b456b7bd" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 41665 } } peers { permanent_uuid: "1d1aad8de5434b16858d1c54bb71922f" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 34049 } }
I20260812 02:32:10.068042 20365 heartbeater.cc:503] Master 127.14.103.62:40947 was elected leader, sending a full tablet report...
I20260812 02:32:10.068500 20386 ts_tablet_manager.cc:1428] T 184a63d17b104e98aaedceddf554fb34 P 1d1aad8de5434b16858d1c54bb71922f: Time spent starting tablet: real 0.007s user 0.004s sys 0.004s
I20260812 02:32:10.230370 20390 raft_consensus.cc:482] T 184a63d17b104e98aaedceddf554fb34 P 5412036532af43dd9fec36a0aa497a41 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 02:32:10.230676 20390 raft_consensus.cc:504] T 184a63d17b104e98aaedceddf554fb34 P 5412036532af43dd9fec36a0aa497a41 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5412036532af43dd9fec36a0aa497a41" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 44115 } } peers { permanent_uuid: "fe6b7b5d2781471da5124683b456b7bd" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 41665 } } peers { permanent_uuid: "1d1aad8de5434b16858d1c54bb71922f" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 34049 } }
I20260812 02:32:10.232014 20390 leader_election.cc:288] T 184a63d17b104e98aaedceddf554fb34 P 5412036532af43dd9fec36a0aa497a41 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe6b7b5d2781471da5124683b456b7bd (127.14.103.2:41665), 1d1aad8de5434b16858d1c54bb71922f (127.14.103.3:34049)
I20260812 02:32:10.234545 20378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:10.232440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.1:57717 (local address 127.14.103.2:41665)
0812 02:32:10.232608 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:10.232612 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:10.232789 (+ 177us) server_negotiation.cc:407] Connection header received
0812 02:32:10.232932 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:10.232937 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:10.232987 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:10.233074 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:32:10.233968 (+ 894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:32:10.233972 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:32:10.233977 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:32:10.233991 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:32:10.234017 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:32:10.234028 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:32:10.234126 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0812 02:32:10.234340 (+ 214us) server_negotiation.cc:299] Negotiation successful
0812 02:32:10.234384 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260812 02:32:10.234548 20393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:10.232240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.2:41665 (local address 127.14.103.1:57717)
0812 02:32:10.232671 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:10.232687 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 02:32:10.232816 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:32:10.233180 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:32:10.233184 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:32:10.233202 (+ 18us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:32:10.233796 (+ 594us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 02:32:10.233798 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:32:10.233810 (+ 12us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 02:32:10.233837 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:32:10.234126 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:32:10.234130 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:32:10.234133 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 02:32:10.234341 (+ 208us) client_negotiation.cc:240] Negotiation successful
0812 02:32:10.234383 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":131,"threads_started":1}
I20260812 02:32:10.234782 20382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:10.232603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.1:32839 (local address 127.14.103.3:34049)
0812 02:32:10.232743 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:10.232749 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:10.233121 (+ 372us) server_negotiation.cc:407] Connection header received
0812 02:32:10.233183 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:10.233186 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:10.233228 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:10.233390 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:32:10.234373 (+ 983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:32:10.234376 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:32:10.234379 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:32:10.234392 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:32:10.234413 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:32:10.234428 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:32:10.234528 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0812 02:32:10.234596 (+ 68us) server_negotiation.cc:299] Negotiation successful
0812 02:32:10.234636 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260812 02:32:10.234802 20394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:10.232418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.3:34049 (local address 127.14.103.1:32839)
0812 02:32:10.232909 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:10.232923 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 02:32:10.233025 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:32:10.233435 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:32:10.233439 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:32:10.233458 (+ 19us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:32:10.234195 (+ 737us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 02:32:10.234197 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:32:10.234206 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 02:32:10.234228 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:32:10.234514 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:32:10.234517 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:32:10.234520 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 02:32:10.234600 (+ 80us) client_negotiation.cc:240] Negotiation successful
0812 02:32:10.234635 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":123,"threads_started":1}
I20260812 02:32:10.235440 20174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "184a63d17b104e98aaedceddf554fb34" candidate_uuid: "5412036532af43dd9fec36a0aa497a41" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe6b7b5d2781471da5124683b456b7bd" is_pre_election: true
I20260812 02:32:10.235445 20318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "184a63d17b104e98aaedceddf554fb34" candidate_uuid: "5412036532af43dd9fec36a0aa497a41" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d1aad8de5434b16858d1c54bb71922f" is_pre_election: true
I20260812 02:32:10.235759 20318 raft_consensus.cc:2454] T 184a63d17b104e98aaedceddf554fb34 P 1d1aad8de5434b16858d1c54bb71922f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5412036532af43dd9fec36a0aa497a41 in term 0.
I20260812 02:32:10.235760 20174 raft_consensus.cc:2454] T 184a63d17b104e98aaedceddf554fb34 P fe6b7b5d2781471da5124683b456b7bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5412036532af43dd9fec36a0aa497a41 in term 0.
I20260812 02:32:10.236320 19955 leader_election.cc:302] T 184a63d17b104e98aaedceddf554fb34 P 5412036532af43dd9fec36a0aa497a41 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1d1aad8de5434b16858d1c54bb71922f, 5412036532af43dd9fec36a0aa497a41, fe6b7b5d2781471da5124683b456b7bd; no voters:
I20260812 02:32:10.236636 20390 raft_consensus.cc:2781] T 184a63d17b104e98aaedceddf554fb34 P 5412036532af43dd9fec36a0aa497a41 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 02:32:10.236721 20390 raft_consensus.cc:482] T 184a63d17b104e98aaedceddf554fb34 P 5412036532af43dd9fec36a0aa497a41 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 02:32:10.236753 20390 raft_consensus.cc:3037] T 184a63d17b104e98aaedceddf554fb34 P 5412036532af43dd9fec36a0aa497a41 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:32:10.237965 20390 raft_consensus.cc:504] T 184a63d17b104e98aaedceddf554fb34 P 5412036532af43dd9fec36a0aa497a41 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5412036532af43dd9fec36a0aa497a41" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 44115 } } peers { permanent_uuid: "fe6b7b5d2781471da5124683b456b7bd" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 41665 } } peers { permanent_uuid: "1d1aad8de5434b16858d1c54bb71922f" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 34049 } }
I20260812 02:32:10.238914 20390 leader_election.cc:288] T 184a63d17b104e98aaedceddf554fb34 P 5412036532af43dd9fec36a0aa497a41 [CANDIDATE]: Term 1 election: Requested vote from peers fe6b7b5d2781471da5124683b456b7bd (127.14.103.2:41665), 1d1aad8de5434b16858d1c54bb71922f (127.14.103.3:34049)
I20260812 02:32:10.239516 20174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "184a63d17b104e98aaedceddf554fb34" candidate_uuid: "5412036532af43dd9fec36a0aa497a41" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe6b7b5d2781471da5124683b456b7bd"
I20260812 02:32:10.239701 20174 raft_consensus.cc:3037] T 184a63d17b104e98aaedceddf554fb34 P fe6b7b5d2781471da5124683b456b7bd [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:32:10.239648 20318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "184a63d17b104e98aaedceddf554fb34" candidate_uuid: "5412036532af43dd9fec36a0aa497a41" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d1aad8de5434b16858d1c54bb71922f"
I20260812 02:32:10.239892 20318 raft_consensus.cc:3037] T 184a63d17b104e98aaedceddf554fb34 P 1d1aad8de5434b16858d1c54bb71922f [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:32:10.241226 20318 raft_consensus.cc:2454] T 184a63d17b104e98aaedceddf554fb34 P 1d1aad8de5434b16858d1c54bb71922f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5412036532af43dd9fec36a0aa497a41 in term 1.
W20260812 02:32:10.241391 20366 tablet.cc:2369] T 184a63d17b104e98aaedceddf554fb34 P 1d1aad8de5434b16858d1c54bb71922f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 02:32:10.241641 19955 leader_election.cc:302] T 184a63d17b104e98aaedceddf554fb34 P 5412036532af43dd9fec36a0aa497a41 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d1aad8de5434b16858d1c54bb71922f, 5412036532af43dd9fec36a0aa497a41; no voters:
I20260812 02:32:10.241904 20390 raft_consensus.cc:2781] T 184a63d17b104e98aaedceddf554fb34 P 5412036532af43dd9fec36a0aa497a41 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:32:10.242224 20390 raft_consensus.cc:686] T 184a63d17b104e98aaedceddf554fb34 P 5412036532af43dd9fec36a0aa497a41 [term 1 LEADER]: Becoming Leader. State: Replica: 5412036532af43dd9fec36a0aa497a41, State: Running, Role: LEADER
I20260812 02:32:10.242064 20174 raft_consensus.cc:2454] T 184a63d17b104e98aaedceddf554fb34 P fe6b7b5d2781471da5124683b456b7bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5412036532af43dd9fec36a0aa497a41 in term 1.
I20260812 02:32:10.242654 20390 consensus_queue.cc:237] T 184a63d17b104e98aaedceddf554fb34 P 5412036532af43dd9fec36a0aa497a41 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5412036532af43dd9fec36a0aa497a41" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 44115 } } peers { permanent_uuid: "fe6b7b5d2781471da5124683b456b7bd" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 41665 } } peers { permanent_uuid: "1d1aad8de5434b16858d1c54bb71922f" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 34049 } }
I20260812 02:32:10.247314 19866 catalog_manager.cc:5495] T 184a63d17b104e98aaedceddf554fb34 P 5412036532af43dd9fec36a0aa497a41 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5412036532af43dd9fec36a0aa497a41 (127.14.103.1). New cstate: current_term: 1 leader_uuid: "5412036532af43dd9fec36a0aa497a41" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5412036532af43dd9fec36a0aa497a41" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 44115 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fe6b7b5d2781471da5124683b456b7bd" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 41665 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1d1aad8de5434b16858d1c54bb71922f" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 34049 } health_report { overall_health: UNKNOWN } } }
W20260812 02:32:10.274626 20224 tablet.cc:2369] T 184a63d17b104e98aaedceddf554fb34 P fe6b7b5d2781471da5124683b456b7bd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 02:32:10.280619 20380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:10.277129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33598 (local address 127.14.103.1:44115)
0812 02:32:10.277455 (+ 326us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:10.277460 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:10.277481 (+ 21us) server_negotiation.cc:407] Connection header received
0812 02:32:10.277708 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:10.277711 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:10.277782 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:10.277916 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:32:10.279394 (+ 1478us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:32:10.279399 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:32:10.279401 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:32:10.279424 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:32:10.279461 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:32:10.279473 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:32:10.279760 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0812 02:32:10.280344 (+ 584us) server_negotiation.cc:299] Negotiation successful
0812 02:32:10.280436 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":191}
I20260812 02:32:10.287684 20318 raft_consensus.cc:1261] T 184a63d17b104e98aaedceddf554fb34 P 1d1aad8de5434b16858d1c54bb71922f [term 1 FOLLOWER]: Refusing update from remote peer 5412036532af43dd9fec36a0aa497a41: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:32:10.288043 20174 raft_consensus.cc:1261] T 184a63d17b104e98aaedceddf554fb34 P fe6b7b5d2781471da5124683b456b7bd [term 1 FOLLOWER]: Refusing update from remote peer 5412036532af43dd9fec36a0aa497a41: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:32:10.288543 20395 consensus_queue.cc:1035] T 184a63d17b104e98aaedceddf554fb34 P 5412036532af43dd9fec36a0aa497a41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d1aad8de5434b16858d1c54bb71922f" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 34049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:32:10.288784 20390 consensus_queue.cc:1035] T 184a63d17b104e98aaedceddf554fb34 P 5412036532af43dd9fec36a0aa497a41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe6b7b5d2781471da5124683b456b7bd" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 41665 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:32:10.299585 20399 mvcc.cc:205] Tried to move back new op lower bound from 7317511906448674816 to 7317511906278850560. Current Snapshot: MvccSnapshot[applied={T|T < 7317511906448674816}]
I20260812 02:32:10.313251 19868 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40024:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 02:32:10.313495 19868 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40024:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 02:32:10.316596 19868 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f2404628f64b4ea897857f98548ce956: Sending DeleteTablet for 3 replicas of tablet 184a63d17b104e98aaedceddf554fb34
I20260812 02:32:10.317646 20005 tablet_service.cc:1514] Processing DeleteTablet for tablet 184a63d17b104e98aaedceddf554fb34 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:32:10 UTC) from {username='slave'} at 127.0.0.1:33596
I20260812 02:32:10.317639 20148 tablet_service.cc:1514] Processing DeleteTablet for tablet 184a63d17b104e98aaedceddf554fb34 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:32:10 UTC) from {username='slave'} at 127.0.0.1:52508
I20260812 02:32:10.317827 20295 tablet_service.cc:1514] Processing DeleteTablet for tablet 184a63d17b104e98aaedceddf554fb34 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:32:10 UTC) from {username='slave'} at 127.0.0.1:35802
I20260812 02:32:10.318250 20409 tablet_replica.cc:330] T 184a63d17b104e98aaedceddf554fb34 P 5412036532af43dd9fec36a0aa497a41: stopping tablet replica
I20260812 02:32:10.318466 20411 tablet_replica.cc:330] T 184a63d17b104e98aaedceddf554fb34 P 1d1aad8de5434b16858d1c54bb71922f: stopping tablet replica
I20260812 02:32:10.318665 20409 raft_consensus.cc:2229] T 184a63d17b104e98aaedceddf554fb34 P 5412036532af43dd9fec36a0aa497a41 [term 1 LEADER]: Raft consensus shutting down.
I20260812 02:32:10.318835 20411 raft_consensus.cc:2229] T 184a63d17b104e98aaedceddf554fb34 P 1d1aad8de5434b16858d1c54bb71922f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 02:32:10.319149 14748 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbep4M4/build/debug/bin/kudu with pid 19934
I20260812 02:32:10.327668 20411 raft_consensus.cc:2258] T 184a63d17b104e98aaedceddf554fb34 P 1d1aad8de5434b16858d1c54bb71922f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:32:10.328287 20410 tablet_replica.cc:330] T 184a63d17b104e98aaedceddf554fb34 P fe6b7b5d2781471da5124683b456b7bd: stopping tablet replica
I20260812 02:32:10.328655 20410 raft_consensus.cc:2229] T 184a63d17b104e98aaedceddf554fb34 P fe6b7b5d2781471da5124683b456b7bd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 02:32:10.329001 20410 raft_consensus.cc:2258] T 184a63d17b104e98aaedceddf554fb34 P fe6b7b5d2781471da5124683b456b7bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:32:10.329772 20411 ts_tablet_manager.cc:1905] T 184a63d17b104e98aaedceddf554fb34 P 1d1aad8de5434b16858d1c54bb71922f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:32:10.330370 20410 ts_tablet_manager.cc:1905] T 184a63d17b104e98aaedceddf554fb34 P fe6b7b5d2781471da5124683b456b7bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:32:10.333154 20410 ts_tablet_manager.cc:1918] T 184a63d17b104e98aaedceddf554fb34 P fe6b7b5d2781471da5124683b456b7bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 02:32:10.333348 20410 log.cc:1196] T 184a63d17b104e98aaedceddf554fb34 P fe6b7b5d2781471da5124683b456b7bd: Deleting WAL directory at /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786501898713699-14748-0/minicluster-data/ts-1/wal/wals/184a63d17b104e98aaedceddf554fb34
I20260812 02:32:10.333554 20411 ts_tablet_manager.cc:1918] T 184a63d17b104e98aaedceddf554fb34 P 1d1aad8de5434b16858d1c54bb71922f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 02:32:10.333678 20411 log.cc:1196] T 184a63d17b104e98aaedceddf554fb34 P 1d1aad8de5434b16858d1c54bb71922f: Deleting WAL directory at /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786501898713699-14748-0/minicluster-data/ts-2/wal/wals/184a63d17b104e98aaedceddf554fb34
I20260812 02:32:10.333768 20410 ts_tablet_manager.cc:1939] T 184a63d17b104e98aaedceddf554fb34 P fe6b7b5d2781471da5124683b456b7bd: Deleting consensus metadata
I20260812 02:32:10.334108 20411 ts_tablet_manager.cc:1939] T 184a63d17b104e98aaedceddf554fb34 P 1d1aad8de5434b16858d1c54bb71922f: Deleting consensus metadata
W20260812 02:32:10.334625 19852 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.103.1:44115 (error 108)
I20260812 02:32:10.335486 19850 catalog_manager.cc:4841] TS fe6b7b5d2781471da5124683b456b7bd (127.14.103.2:41665): tablet 184a63d17b104e98aaedceddf554fb34 (table test-table [id=e1f4ddc3f78a497690e089b390fd142e]) successfully deleted
I20260812 02:32:10.335486 19854 catalog_manager.cc:4841] TS 1d1aad8de5434b16858d1c54bb71922f (127.14.103.3:34049): tablet 184a63d17b104e98aaedceddf554fb34 (table test-table [id=e1f4ddc3f78a497690e089b390fd142e]) successfully deleted
I20260812 02:32:10.336177 20377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:10.335893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.1:44115 (local address 127.0.0.1:33600)
0812 02:32:10.336025 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:10.336096 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.1:44115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260812 02:32:10.336303 14748 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbep4M4/build/debug/bin/kudu with pid 20080
W20260812 02:32:10.336433 19852 catalog_manager.cc:4584] TS 5412036532af43dd9fec36a0aa497a41 (127.14.103.1:44115): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 184a63d17b104e98aaedceddf554fb34: Network error: Client connection negotiation failed: client connection to 127.14.103.1:44115: connect: Connection refused (error 111)
I20260812 02:32:10.345466 14748 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbep4M4/build/debug/bin/kudu with pid 20226
I20260812 02:32:10.353690 14748 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbep4M4/build/debug/bin/kudu with pid 19829
2026-08-12T02:32:10Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (1323 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260812 02:32:10.380348 14748 mini_oidc.cc:152] Starting JWKS server
I20260812 02:32:10.382494 14748 webserver.cc:460] Webserver started at https://127.0.0.1:34321/ using document root <none> and password file <none>
I20260812 02:32:10.382715 14748 mini_oidc.cc:189] Starting OIDC Discovery server
I20260812 02:32:10.383163 14748 webserver.cc:460] Webserver started at http://127.0.0.1:41655/ using document root <none> and password file <none>
2026-08-12T02:32:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T02:32:10Z Disabled control of system clock
I20260812 02:32:10.396169 14748 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbep4M4/build/debug/bin/kudu
/tmp/dist-test-taskbep4M4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786501898713699-14748-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786501898713699-14748-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786501898713699-14748-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786501898713699-14748-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.103.62:42847
--webserver_interface=127.14.103.62
--webserver_port=0
--builtin_ntp_servers=127.14.103.20:40649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.103.62:42847
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34321/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786501898713699-14748-0/testchainca.cert with env {}
W20260812 02:32:10.525342 20419 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:32:10.525660 20419 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:32:10.525707 20419 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:32:10.530026 20419 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:32:10.530117 20419 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:32:10.530133 20419 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:32:10.530151 20419 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:32:10.530167 20419 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260812 02:32:10.530184 20419 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260812 02:32:10.530198 20419 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:34321/jwks.json
I20260812 02:32:10.536679 20419 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.103.20:40649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786501898713699-14748-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786501898713699-14748-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.103.62:42847
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.103.62:42847
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34321/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786501898713699-14748-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.103.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786501898713699-14748-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20419
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786501898713699-14748-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:32:10.538053 20419 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:32:10.539388 20419 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:32:10.549109 20430 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:32:10.549391 20419 server_base.cc:1034] running on GCE node
W20260812 02:32:10.549275 20427 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:32:10.549109 20428 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:32:10.550535 20419 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:32:10.551887 20419 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:32:10.553128 20419 hybrid_clock.cc:648] HybridClock initialized: now 1786501930553100 us; error 49 us; skew 500 ppm
I20260812 02:32:10.555368 20419 webserver.cc:460] Webserver started at http://127.14.103.62:40707/ using document root <none> and password file <none>
I20260812 02:32:10.556010 20419 fs_manager.cc:359] Metadata directory not provided
I20260812 02:32:10.556113 20419 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:32:10.556365 20419 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:32:10.558739 20419 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786501898713699-14748-0/minicluster-data/master-0/data/instance:
uuid: "e7491e41fbd64b4da499c07dd8886049"
format_stamp: "Formatted at 2026-08-12 02:32:10 on dist-test-slave-2w3w"
server_key: "5d6fea4c1bdd23b695d318da2827649e"
server_key_iv: "531306bb757b1f6215608bd2530044d3"
server_key_version: "encryptionkey@0"
I20260812 02:32:10.559508 20419 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786501898713699-14748-0/minicluster-data/master-0/wal/instance:
uuid: "e7491e41fbd64b4da499c07dd8886049"
format_stamp: "Formatted at 2026-08-12 02:32:10 on dist-test-slave-2w3w"
server_key: "5d6fea4c1bdd23b695d318da2827649e"
server_key_iv: "531306bb757b1f6215608bd2530044d3"
server_key_version: "encryptionkey@0"
I20260812 02:32:10.563886 20419 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260812 02:32:10.567382 20436 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:32:10.568841 20419 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260812 02:32:10.569038 20419 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786501898713699-14748-0/minicluster-data/master-0/data,/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786501898713699-14748-0/minicluster-data/master-0/wal
uuid: "e7491e41fbd64b4da499c07dd8886049"
format_stamp: "Formatted at 2026-08-12 02:32:10 on dist-test-slave-2w3w"
server_key: "5d6fea4c1bdd23b695d318da2827649e"
server_key_iv: "531306bb757b1f6215608bd2530044d3"
server_key_version: "encryptionkey@0"
I20260812 02:32:10.569182 20419 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786501898713699-14748-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786501898713699-14748-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786501898713699-14748-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:32:10.592814 20419 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:32:10.593750 20419 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:32:10.593995 20419 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:32:10.601783 20419 rpc_server.cc:307] RPC server started. Bound to: 127.14.103.62:42847
I20260812 02:32:10.601810 20494 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.103.62:42847 every 8 connection(s)
I20260812 02:32:10.602830 20419 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786501898713699-14748-0/minicluster-data/master-0/data/info.pb
I20260812 02:32:10.605046 14748 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbep4M4/build/debug/bin/kudu as pid 20419
I20260812 02:32:10.605203 14748 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786501898713699-14748-0/minicluster-data/master-0/wal/instance
I20260812 02:32:10.605520 14748 external_mini_cluster.cc:1428] Setting key 7745c06631f7099cbff932f0020d4eb4
I20260812 02:32:10.607127 20495 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:32:10.613485 20495 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7491e41fbd64b4da499c07dd8886049: Bootstrap starting.
I20260812 02:32:10.616060 20498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:10.607137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57472 (local address 127.14.103.62:42847)
0812 02:32:10.607951 (+ 814us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:10.607969 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:10.608022 (+ 53us) server_negotiation.cc:407] Connection header received
0812 02:32:10.608708 (+ 686us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:10.608726 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:10.609158 (+ 432us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:10.609577 (+ 419us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:32:10.610050 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:10.611284 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:10.612280 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:10.612812 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:10.613007 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:10.614057 (+ 1050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:32:10.614063 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:32:10.614077 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:32:10.614500 (+ 423us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:32:10.614571 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:32:10.614593 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:32:10.614713 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0812 02:32:10.614916 (+ 203us) server_negotiation.cc:299] Negotiation successful
0812 02:32:10.615273 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":407,"thread_start_us":195,"threads_started":1}
I20260812 02:32:10.616128 20495 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e7491e41fbd64b4da499c07dd8886049: Neither blocks nor log segments found. Creating new log.
I20260812 02:32:10.617025 20495 log.cc:824] T 00000000000000000000000000000000 P e7491e41fbd64b4da499c07dd8886049: Log is configured to *not* fsync() on all Append() calls
I20260812 02:32:10.619614 14748 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 02:32:10.619916 20495 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7491e41fbd64b4da499c07dd8886049: No bootstrap required, opened a new log
I20260812 02:32:10.624155 20495 raft_consensus.cc:348] T 00000000000000000000000000000000 P e7491e41fbd64b4da499c07dd8886049 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7491e41fbd64b4da499c07dd8886049" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 42847 } }
I20260812 02:32:10.624472 20495 raft_consensus.cc:374] T 00000000000000000000000000000000 P e7491e41fbd64b4da499c07dd8886049 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:32:10.624511 20495 raft_consensus.cc:729] T 00000000000000000000000000000000 P e7491e41fbd64b4da499c07dd8886049 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7491e41fbd64b4da499c07dd8886049, State: Initialized, Role: FOLLOWER
I20260812 02:32:10.625002 20495 consensus_queue.cc:260] T 00000000000000000000000000000000 P e7491e41fbd64b4da499c07dd8886049 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7491e41fbd64b4da499c07dd8886049" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 42847 } }
I20260812 02:32:10.625167 20495 raft_consensus.cc:388] T 00000000000000000000000000000000 P e7491e41fbd64b4da499c07dd8886049 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 02:32:10.625216 20495 raft_consensus.cc:482] T 00000000000000000000000000000000 P e7491e41fbd64b4da499c07dd8886049 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 02:32:10.625293 20495 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e7491e41fbd64b4da499c07dd8886049 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:32:10.626466 20495 raft_consensus.cc:504] T 00000000000000000000000000000000 P e7491e41fbd64b4da499c07dd8886049 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7491e41fbd64b4da499c07dd8886049" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 42847 } }
I20260812 02:32:10.626935 20495 leader_election.cc:302] T 00000000000000000000000000000000 P e7491e41fbd64b4da499c07dd8886049 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e7491e41fbd64b4da499c07dd8886049; no voters:
I20260812 02:32:10.627465 20495 leader_election.cc:288] T 00000000000000000000000000000000 P e7491e41fbd64b4da499c07dd8886049 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 02:32:10.627837 20501 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e7491e41fbd64b4da499c07dd8886049 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:32:10.628322 20501 raft_consensus.cc:686] T 00000000000000000000000000000000 P e7491e41fbd64b4da499c07dd8886049 [term 1 LEADER]: Becoming Leader. State: Replica: e7491e41fbd64b4da499c07dd8886049, State: Running, Role: LEADER
I20260812 02:32:10.629123 20501 consensus_queue.cc:237] T 00000000000000000000000000000000 P e7491e41fbd64b4da499c07dd8886049 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7491e41fbd64b4da499c07dd8886049" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 42847 } }
I20260812 02:32:10.629549 20495 sys_catalog.cc:564] T 00000000000000000000000000000000 P e7491e41fbd64b4da499c07dd8886049 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:32:10.631453 20502 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7491e41fbd64b4da499c07dd8886049 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e7491e41fbd64b4da499c07dd8886049. Latest consensus state: current_term: 1 leader_uuid: "e7491e41fbd64b4da499c07dd8886049" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7491e41fbd64b4da499c07dd8886049" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 42847 } } }
I20260812 02:32:10.631516 20503 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7491e41fbd64b4da499c07dd8886049 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e7491e41fbd64b4da499c07dd8886049" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7491e41fbd64b4da499c07dd8886049" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 42847 } } }
I20260812 02:32:10.631670 20502 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7491e41fbd64b4da499c07dd8886049 [sys.catalog]: This master's current role is: LEADER
I20260812 02:32:10.631680 20503 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7491e41fbd64b4da499c07dd8886049 [sys.catalog]: This master's current role is: LEADER
W20260812 02:32:10.635108 20518 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e7491e41fbd64b4da499c07dd8886049: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 02:32:10.635198 20518 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 02:32:10.635278 20517 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 02:32:10.636006 20517 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 02:32:10.642508 20517 catalog_manager.cc:1347] Generated new cluster ID: 932e0c67b2374c54b92140c735cc18f2
I20260812 02:32:10.642625 20517 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 02:32:10.668843 20517 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 02:32:10.670295 20517 catalog_manager.cc:1504] Loading token signing keys...
I20260812 02:32:10.677908 20517 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e7491e41fbd64b4da499c07dd8886049: Generated new TSK 0
I20260812 02:32:10.679131 20517 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260812 02:32:10.719544 20530 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260812 02:32:10.719787 20530 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260812 02:32:10.720331 20498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:10.701157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57478 (local address 127.14.103.62:42847)
0812 02:32:10.701332 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:10.701337 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:10.701540 (+ 203us) server_negotiation.cc:407] Connection header received
0812 02:32:10.701767 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:10.701770 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:10.701835 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:10.701963 (+ 128us) server_negotiation.cc:226] Negotiated authn=JWT
0812 02:32:10.702252 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:10.703125 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:10.704108 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:10.704475 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:10.704589 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:10.705048 (+ 459us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0812 02:32:10.719871 (+ 14823us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34321/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0812 02:32:10.720171 (+ 300us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:57478: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34321/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":61}
W20260812 02:32:10.720424 20498 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:57478: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34321/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260812 02:32:10.721735 20529 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:32:10.701002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:42847 (local address 127.0.0.1:57478)
0812 02:32:10.701443 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:10.701458 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 02:32:10.701588 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:32:10.702008 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:32:10.702011 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:32:10.702018 (+ 7us) client_negotiation.cc:189] Negotiated authn=JWT
0812 02:32:10.702128 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:10.702133 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:10.703276 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:10.703280 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:10.703973 (+ 693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:10.703983 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:10.704674 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:10.704678 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:10.704758 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:10.704816 (+ 58us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0812 02:32:10.720140 (+ 15324us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34321/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0812 02:32:10.720371 (+ 231us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.14.103.62:42847: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34321/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":137,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.14.103.62:42847: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34321/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260812 02:32:10.722637 14748 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbep4M4/build/debug/bin/kudu with pid 20419
2026-08-12T02:32:10Z chronyd exiting
I20260812 02:32:10.763434 14748 test_util.cc:175] -----------------------------------------------
I20260812 02:32:10.763581 14748 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786501898713699-14748-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (385 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260812 02:32:10.764966 14748 mini_oidc.cc:152] Starting JWKS server
I20260812 02:32:10.765905 14748 webserver.cc:460] Webserver started at https://127.0.0.1:38941/ using document root <none> and password file <none>
I20260812 02:32:10.766045 14748 mini_oidc.cc:189] Starting OIDC Discovery server
I20260812 02:32:10.766355 14748 webserver.cc:460] Webserver started at http://127.0.0.1:39107/ using document root <none> and password file <none>
2026-08-12T02:32:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T02:32:10Z Disabled control of system clock
I20260812 02:32:10.778514 14748 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbep4M4/build/debug/bin/kudu
/tmp/dist-test-taskbep4M4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786501898713699-14748-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786501898713699-14748-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786501898713699-14748-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786501898713699-14748-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.103.62:45801
--webserver_interface=127.14.103.62
--webserver_port=0
--builtin_ntp_servers=127.14.103.20:32925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.103.62:45801
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38941/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786501898713699-14748-0/testchainca.cert with env {}
W20260812 02:32:10.905157 20538 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:32:10.905503 20538 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:32:10.905550 20538 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:32:10.909875 20538 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:32:10.909981 20538 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:32:10.909997 20538 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:32:10.910015 20538 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:32:10.910030 20538 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260812 02:32:10.910048 20538 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260812 02:32:10.910061 20538 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:38941/jwks.json
I20260812 02:32:10.916067 20538 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.103.20:32925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786501898713699-14748-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786501898713699-14748-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.103.62:45801
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.103.62:45801
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38941/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786501898713699-14748-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.103.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786501898713699-14748-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20538
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786501898713699-14748-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:32:10.917225 20538 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:32:10.918275 20538 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:32:10.928007 20545 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:32:10.927994 20546 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:32:10.928275 20538 server_base.cc:1034] running on GCE node
W20260812 02:32:10.927994 20548 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:32:10.929136 20538 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:32:10.930408 20538 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:32:10.931635 20538 hybrid_clock.cc:648] HybridClock initialized: now 1786501930931605 us; error 57 us; skew 500 ppm
I20260812 02:32:10.933959 20538 webserver.cc:460] Webserver started at http://127.14.103.62:40719/ using document root <none> and password file <none>
I20260812 02:32:10.934644 20538 fs_manager.cc:359] Metadata directory not provided
I20260812 02:32:10.934772 20538 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:32:10.935094 20538 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:32:10.937630 20538 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786501898713699-14748-0/minicluster-data/master-0/data/instance:
uuid: "da6acf326acb481ab4db80a2c61755d2"
format_stamp: "Formatted at 2026-08-12 02:32:10 on dist-test-slave-2w3w"
server_key: "83d9789d58beac1166cf8e78d10a137c"
server_key_iv: "74309395aab5cf4cffdce54b40d83554"
server_key_version: "encryptionkey@0"
I20260812 02:32:10.938422 20538 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786501898713699-14748-0/minicluster-data/master-0/wal/instance:
uuid: "da6acf326acb481ab4db80a2c61755d2"
format_stamp: "Formatted at 2026-08-12 02:32:10 on dist-test-slave-2w3w"
server_key: "83d9789d58beac1166cf8e78d10a137c"
server_key_iv: "74309395aab5cf4cffdce54b40d83554"
server_key_version: "encryptionkey@0"
I20260812 02:32:10.943521 20538 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260812 02:32:10.946780 20554 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:32:10.948513 20538 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 02:32:10.948706 20538 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786501898713699-14748-0/minicluster-data/master-0/data,/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786501898713699-14748-0/minicluster-data/master-0/wal
uuid: "da6acf326acb481ab4db80a2c61755d2"
format_stamp: "Formatted at 2026-08-12 02:32:10 on dist-test-slave-2w3w"
server_key: "83d9789d58beac1166cf8e78d10a137c"
server_key_iv: "74309395aab5cf4cffdce54b40d83554"
server_key_version: "encryptionkey@0"
I20260812 02:32:10.948858 20538 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786501898713699-14748-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786501898713699-14748-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786501898713699-14748-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:32:10.971928 20538 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:32:10.972747 20538 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:32:10.972988 20538 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:32:10.981359 20538 rpc_server.cc:307] RPC server started. Bound to: 127.14.103.62:45801
I20260812 02:32:10.981379 20614 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.103.62:45801 every 8 connection(s)
I20260812 02:32:10.982514 20538 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786501898713699-14748-0/minicluster-data/master-0/data/info.pb
I20260812 02:32:10.986573 20616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:32:10.987044 14748 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbep4M4/build/debug/bin/kudu as pid 20538
I20260812 02:32:10.987170 14748 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786501898713699-14748-0/minicluster-data/master-0/wal/instance
I20260812 02:32:10.987515 14748 external_mini_cluster.cc:1428] Setting key a9f352b77294863b4ce5a452fb203956
I20260812 02:32:10.995055 20616 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da6acf326acb481ab4db80a2c61755d2: Bootstrap starting.
I20260812 02:32:10.997751 20616 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P da6acf326acb481ab4db80a2c61755d2: Neither blocks nor log segments found. Creating new log.
I20260812 02:32:10.998811 20616 log.cc:824] T 00000000000000000000000000000000 P da6acf326acb481ab4db80a2c61755d2: Log is configured to *not* fsync() on all Append() calls
I20260812 02:32:10.999686 20619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:10.989144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58898 (local address 127.14.103.62:45801)
0812 02:32:10.989967 (+ 823us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:10.989986 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:10.990062 (+ 76us) server_negotiation.cc:407] Connection header received
0812 02:32:10.991197 (+ 1135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:10.991229 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:10.991809 (+ 580us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:10.992287 (+ 478us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:32:10.992783 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:10.994187 (+ 1404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:10.995302 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:10.996112 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:10.996335 (+ 223us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:10.997454 (+ 1119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:32:10.997459 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:32:10.997474 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:32:10.997951 (+ 477us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:32:10.998046 (+ 95us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:32:10.998069 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:32:10.998296 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0812 02:32:10.998415 (+ 119us) server_negotiation.cc:299] Negotiation successful
0812 02:32:10.998711 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":367,"thread_start_us":149,"threads_started":1}
I20260812 02:32:11.001840 20616 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da6acf326acb481ab4db80a2c61755d2: No bootstrap required, opened a new log
I20260812 02:32:11.003233 14748 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 02:32:11.007372 20616 raft_consensus.cc:348] T 00000000000000000000000000000000 P da6acf326acb481ab4db80a2c61755d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da6acf326acb481ab4db80a2c61755d2" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 45801 } }
I20260812 02:32:11.007818 20616 raft_consensus.cc:374] T 00000000000000000000000000000000 P da6acf326acb481ab4db80a2c61755d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:32:11.007929 20616 raft_consensus.cc:729] T 00000000000000000000000000000000 P da6acf326acb481ab4db80a2c61755d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da6acf326acb481ab4db80a2c61755d2, State: Initialized, Role: FOLLOWER
I20260812 02:32:11.008441 20616 consensus_queue.cc:260] T 00000000000000000000000000000000 P da6acf326acb481ab4db80a2c61755d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da6acf326acb481ab4db80a2c61755d2" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 45801 } }
I20260812 02:32:11.008589 20616 raft_consensus.cc:388] T 00000000000000000000000000000000 P da6acf326acb481ab4db80a2c61755d2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 02:32:11.008680 20616 raft_consensus.cc:482] T 00000000000000000000000000000000 P da6acf326acb481ab4db80a2c61755d2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 02:32:11.008771 20616 raft_consensus.cc:3037] T 00000000000000000000000000000000 P da6acf326acb481ab4db80a2c61755d2 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:32:11.009917 20616 raft_consensus.cc:504] T 00000000000000000000000000000000 P da6acf326acb481ab4db80a2c61755d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da6acf326acb481ab4db80a2c61755d2" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 45801 } }
I20260812 02:32:11.010380 20616 leader_election.cc:302] T 00000000000000000000000000000000 P da6acf326acb481ab4db80a2c61755d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: da6acf326acb481ab4db80a2c61755d2; no voters:
I20260812 02:32:11.010815 20616 leader_election.cc:288] T 00000000000000000000000000000000 P da6acf326acb481ab4db80a2c61755d2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 02:32:11.011063 20621 raft_consensus.cc:2781] T 00000000000000000000000000000000 P da6acf326acb481ab4db80a2c61755d2 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:32:11.011322 20621 raft_consensus.cc:686] T 00000000000000000000000000000000 P da6acf326acb481ab4db80a2c61755d2 [term 1 LEADER]: Becoming Leader. State: Replica: da6acf326acb481ab4db80a2c61755d2, State: Running, Role: LEADER
I20260812 02:32:11.011847 20621 consensus_queue.cc:237] T 00000000000000000000000000000000 P da6acf326acb481ab4db80a2c61755d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da6acf326acb481ab4db80a2c61755d2" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 45801 } }
I20260812 02:32:11.012315 20616 sys_catalog.cc:564] T 00000000000000000000000000000000 P da6acf326acb481ab4db80a2c61755d2 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:32:11.013674 20622 sys_catalog.cc:455] T 00000000000000000000000000000000 P da6acf326acb481ab4db80a2c61755d2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "da6acf326acb481ab4db80a2c61755d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da6acf326acb481ab4db80a2c61755d2" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 45801 } } }
I20260812 02:32:11.013975 20622 sys_catalog.cc:458] T 00000000000000000000000000000000 P da6acf326acb481ab4db80a2c61755d2 [sys.catalog]: This master's current role is: LEADER
I20260812 02:32:11.014202 20623 sys_catalog.cc:455] T 00000000000000000000000000000000 P da6acf326acb481ab4db80a2c61755d2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader da6acf326acb481ab4db80a2c61755d2. Latest consensus state: current_term: 1 leader_uuid: "da6acf326acb481ab4db80a2c61755d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da6acf326acb481ab4db80a2c61755d2" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 45801 } } }
I20260812 02:32:11.014392 20623 sys_catalog.cc:458] T 00000000000000000000000000000000 P da6acf326acb481ab4db80a2c61755d2 [sys.catalog]: This master's current role is: LEADER
W20260812 02:32:11.017689 20638 catalog_manager.cc:1558] T 00000000000000000000000000000000 P da6acf326acb481ab4db80a2c61755d2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 02:32:11.017799 20638 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 02:32:11.017879 20634 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 02:32:11.018652 20634 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 02:32:11.025064 20634 catalog_manager.cc:1347] Generated new cluster ID: 93af993a8eb0400197766cbac20e23f5
I20260812 02:32:11.025274 20634 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 02:32:11.035871 20634 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 02:32:11.036974 20634 catalog_manager.cc:1504] Loading token signing keys...
I20260812 02:32:11.042418 20634 catalog_manager.cc:5868] T 00000000000000000000000000000000 P da6acf326acb481ab4db80a2c61755d2: Generated new TSK 0
I20260812 02:32:11.043257 20634 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260812 02:32:11.088786 20649 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260812 02:32:11.088902 20649 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260812 02:32:11.089534 20619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:11.061714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58900 (local address 127.14.103.62:45801)
0812 02:32:11.061866 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:11.061870 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:11.062083 (+ 213us) server_negotiation.cc:407] Connection header received
0812 02:32:11.062196 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:11.062198 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:11.062255 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:11.062392 (+ 137us) server_negotiation.cc:226] Negotiated authn=JWT
0812 02:32:11.062740 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:11.063738 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:11.065394 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:11.065671 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:11.065779 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:11.066420 (+ 641us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0812 02:32:11.089083 (+ 22663us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38941/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0812 02:32:11.089402 (+ 319us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:58900: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38941/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":60}
W20260812 02:32:11.089627 20619 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:58900: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38941/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260812 02:32:11.089758 20648 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:32:11.061459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:45801 (local address 127.0.0.1:58900)
0812 02:32:11.061870 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:11.061887 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 02:32:11.062026 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:32:11.062416 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:32:11.062420 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:32:11.062428 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0812 02:32:11.062569 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:11.062576 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:11.063913 (+ 1337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:11.063920 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:11.065097 (+ 1177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:11.065235 (+ 138us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:11.065834 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:11.065839 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:11.065971 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:11.066014 (+ 43us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0812 02:32:11.089347 (+ 23333us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38941/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0812 02:32:11.089577 (+ 230us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.14.103.62:45801: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38941/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":125,"threads_started":1}
I20260812 02:32:11.090734 14748 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbep4M4/build/debug/bin/kudu with pid 20538
2026-08-12T02:32:11Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (373 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260812 02:32:11.138190 14748 mini_oidc.cc:152] Starting JWKS server
I20260812 02:32:11.139233 14748 webserver.cc:460] Webserver started at https://127.0.0.1:36483/ using document root <none> and password file <none>
I20260812 02:32:11.139379 14748 mini_oidc.cc:189] Starting OIDC Discovery server
I20260812 02:32:11.139653 14748 webserver.cc:460] Webserver started at http://127.0.0.1:35651/ using document root <none> and password file <none>
2026-08-12T02:32:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T02:32:11Z Disabled control of system clock
I20260812 02:32:11.152660 14748 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbep4M4/build/debug/bin/kudu
/tmp/dist-test-taskbep4M4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786501898713699-14748-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786501898713699-14748-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786501898713699-14748-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786501898713699-14748-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.103.62:43501
--webserver_interface=127.14.103.62
--webserver_port=0
--builtin_ntp_servers=127.14.103.20:43717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.103.62:43501
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36483/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260812 02:32:11.281522 20656 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:32:11.281872 20656 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:32:11.281924 20656 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:32:11.286531 20656 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:32:11.286615 20656 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:32:11.286631 20656 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:32:11.286649 20656 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:32:11.286665 20656 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260812 02:32:11.286682 20656 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260812 02:32:11.286696 20656 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:36483/jwks.json
I20260812 02:32:11.293144 20656 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.103.20:43717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786501898713699-14748-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786501898713699-14748-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.103.62:43501
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.103.62:43501
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36483/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786501898713699-14748-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.103.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20656
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786501898713699-14748-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:32:11.294413 20656 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:32:11.295981 20656 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:32:11.306118 20666 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:32:11.306258 20662 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:32:11.306348 20656 server_base.cc:1034] running on GCE node
W20260812 02:32:11.306348 20663 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:32:11.307211 20656 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:32:11.308990 20656 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:32:11.310282 20656 hybrid_clock.cc:648] HybridClock initialized: now 1786501931310239 us; error 35 us; skew 500 ppm
I20260812 02:32:11.312701 20656 webserver.cc:460] Webserver started at http://127.14.103.62:43555/ using document root <none> and password file <none>
I20260812 02:32:11.313339 20656 fs_manager.cc:359] Metadata directory not provided
I20260812 02:32:11.313444 20656 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:32:11.313653 20656 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:32:11.316246 20656 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786501898713699-14748-0/minicluster-data/master-0/data/instance:
uuid: "60e0c2de44d245b1b624e96543092c52"
format_stamp: "Formatted at 2026-08-12 02:32:11 on dist-test-slave-2w3w"
server_key: "2bf92b601fd6f3671870eeaa378701fd"
server_key_iv: "d8f5a82d5767817e35577e0d271b9d8e"
server_key_version: "encryptionkey@0"
I20260812 02:32:11.316944 20656 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786501898713699-14748-0/minicluster-data/master-0/wal/instance:
uuid: "60e0c2de44d245b1b624e96543092c52"
format_stamp: "Formatted at 2026-08-12 02:32:11 on dist-test-slave-2w3w"
server_key: "2bf92b601fd6f3671870eeaa378701fd"
server_key_iv: "d8f5a82d5767817e35577e0d271b9d8e"
server_key_version: "encryptionkey@0"
I20260812 02:32:11.322690 20656 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260812 02:32:11.325978 20672 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:32:11.327639 20656 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 02:32:11.327832 20656 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786501898713699-14748-0/minicluster-data/master-0/data,/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786501898713699-14748-0/minicluster-data/master-0/wal
uuid: "60e0c2de44d245b1b624e96543092c52"
format_stamp: "Formatted at 2026-08-12 02:32:11 on dist-test-slave-2w3w"
server_key: "2bf92b601fd6f3671870eeaa378701fd"
server_key_iv: "d8f5a82d5767817e35577e0d271b9d8e"
server_key_version: "encryptionkey@0"
I20260812 02:32:11.327983 20656 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786501898713699-14748-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786501898713699-14748-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786501898713699-14748-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:32:11.351459 20656 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:32:11.352401 20656 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:32:11.352588 20656 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:32:11.360514 20656 rpc_server.cc:307] RPC server started. Bound to: 127.14.103.62:43501
I20260812 02:32:11.360525 20731 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.103.62:43501 every 8 connection(s)
I20260812 02:32:11.361574 20656 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786501898713699-14748-0/minicluster-data/master-0/data/info.pb
I20260812 02:32:11.362003 14748 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbep4M4/build/debug/bin/kudu as pid 20656
I20260812 02:32:11.362112 14748 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786501898713699-14748-0/minicluster-data/master-0/wal/instance
I20260812 02:32:11.362563 14748 external_mini_cluster.cc:1428] Setting key 01d3014a35fcd94d325ac4801dad2bd7
I20260812 02:32:11.365283 20732 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:32:11.372727 20732 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60e0c2de44d245b1b624e96543092c52: Bootstrap starting.
I20260812 02:32:11.374109 20735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:11.364173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58850 (local address 127.14.103.62:43501)
0812 02:32:11.364894 (+ 721us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:11.364922 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:11.364960 (+ 38us) server_negotiation.cc:407] Connection header received
0812 02:32:11.365689 (+ 729us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:11.365708 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:11.366121 (+ 413us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:11.366423 (+ 302us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:32:11.366845 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:11.367940 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:11.369523 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:11.370480 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:11.370636 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:11.371678 (+ 1042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:32:11.371684 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:32:11.371700 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:32:11.372360 (+ 660us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:32:11.372485 (+ 125us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:32:11.372527 (+ 42us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:32:11.372792 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0812 02:32:11.372954 (+ 162us) server_negotiation.cc:299] Negotiation successful
0812 02:32:11.373306 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":172,"threads_started":1}
I20260812 02:32:11.375468 20732 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 60e0c2de44d245b1b624e96543092c52: Neither blocks nor log segments found. Creating new log.
I20260812 02:32:11.376427 20732 log.cc:824] T 00000000000000000000000000000000 P 60e0c2de44d245b1b624e96543092c52: Log is configured to *not* fsync() on all Append() calls
I20260812 02:32:11.377686 14748 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 02:32:11.379581 20732 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60e0c2de44d245b1b624e96543092c52: No bootstrap required, opened a new log
I20260812 02:32:11.383816 20732 raft_consensus.cc:348] T 00000000000000000000000000000000 P 60e0c2de44d245b1b624e96543092c52 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60e0c2de44d245b1b624e96543092c52" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 43501 } }
I20260812 02:32:11.384168 20732 raft_consensus.cc:374] T 00000000000000000000000000000000 P 60e0c2de44d245b1b624e96543092c52 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:32:11.384248 20732 raft_consensus.cc:729] T 00000000000000000000000000000000 P 60e0c2de44d245b1b624e96543092c52 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60e0c2de44d245b1b624e96543092c52, State: Initialized, Role: FOLLOWER
I20260812 02:32:11.384752 20732 consensus_queue.cc:260] T 00000000000000000000000000000000 P 60e0c2de44d245b1b624e96543092c52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60e0c2de44d245b1b624e96543092c52" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 43501 } }
I20260812 02:32:11.384939 20732 raft_consensus.cc:388] T 00000000000000000000000000000000 P 60e0c2de44d245b1b624e96543092c52 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 02:32:11.385042 20732 raft_consensus.cc:482] T 00000000000000000000000000000000 P 60e0c2de44d245b1b624e96543092c52 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 02:32:11.385181 20732 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 60e0c2de44d245b1b624e96543092c52 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:32:11.387116 20732 raft_consensus.cc:504] T 00000000000000000000000000000000 P 60e0c2de44d245b1b624e96543092c52 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60e0c2de44d245b1b624e96543092c52" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 43501 } }
I20260812 02:32:11.387516 20732 leader_election.cc:302] T 00000000000000000000000000000000 P 60e0c2de44d245b1b624e96543092c52 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 60e0c2de44d245b1b624e96543092c52; no voters:
I20260812 02:32:11.387866 20732 leader_election.cc:288] T 00000000000000000000000000000000 P 60e0c2de44d245b1b624e96543092c52 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 02:32:11.388130 20738 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 60e0c2de44d245b1b624e96543092c52 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:32:11.388559 20738 raft_consensus.cc:686] T 00000000000000000000000000000000 P 60e0c2de44d245b1b624e96543092c52 [term 1 LEADER]: Becoming Leader. State: Replica: 60e0c2de44d245b1b624e96543092c52, State: Running, Role: LEADER
I20260812 02:32:11.388950 20738 consensus_queue.cc:237] T 00000000000000000000000000000000 P 60e0c2de44d245b1b624e96543092c52 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60e0c2de44d245b1b624e96543092c52" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 43501 } }
I20260812 02:32:11.389662 20732 sys_catalog.cc:564] T 00000000000000000000000000000000 P 60e0c2de44d245b1b624e96543092c52 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:32:11.390906 20739 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60e0c2de44d245b1b624e96543092c52 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "60e0c2de44d245b1b624e96543092c52" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60e0c2de44d245b1b624e96543092c52" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 43501 } } }
I20260812 02:32:11.391117 20739 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60e0c2de44d245b1b624e96543092c52 [sys.catalog]: This master's current role is: LEADER
I20260812 02:32:11.391449 20740 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60e0c2de44d245b1b624e96543092c52 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 60e0c2de44d245b1b624e96543092c52. Latest consensus state: current_term: 1 leader_uuid: "60e0c2de44d245b1b624e96543092c52" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60e0c2de44d245b1b624e96543092c52" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 43501 } } }
I20260812 02:32:11.391924 20740 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60e0c2de44d245b1b624e96543092c52 [sys.catalog]: This master's current role is: LEADER
I20260812 02:32:11.393146 20747 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 02:32:11.396557 20747 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 02:32:11.409337 20747 catalog_manager.cc:1347] Generated new cluster ID: 25fdb12a7bcd46fa8bda85650fa8674b
I20260812 02:32:11.409555 20747 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 02:32:11.437070 20747 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 02:32:11.438441 20747 catalog_manager.cc:1504] Loading token signing keys...
I20260812 02:32:11.449054 20747 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 60e0c2de44d245b1b624e96543092c52: Generated new TSK 0
I20260812 02:32:11.450101 20747 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 02:32:11.477047 20735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:11.467677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58852 (local address 127.14.103.62:43501)
0812 02:32:11.467849 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:11.467854 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:11.468164 (+ 310us) server_negotiation.cc:407] Connection header received
0812 02:32:11.468358 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:11.468362 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:11.468436 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:11.468577 (+ 141us) server_negotiation.cc:226] Negotiated authn=JWT
0812 02:32:11.468894 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:11.469766 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:11.471829 (+ 2063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:11.472215 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:11.472372 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:11.472976 (+ 604us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0812 02:32:11.476317 (+ 3341us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36483/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0812 02:32:11.476832 (+ 515us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:58852: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36483/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":84}
W20260812 02:32:11.477212 20735 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:58852: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36483/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260812 02:32:11.478118 20766 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:32:11.467562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:43501 (local address 127.0.0.1:58852)
0812 02:32:11.468033 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:11.468048 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 02:32:11.468183 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:32:11.468636 (+ 453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:32:11.468639 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:32:11.468647 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0812 02:32:11.468761 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:11.468766 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:11.470158 (+ 1392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:11.470163 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:11.471539 (+ 1376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:11.471567 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:11.472538 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:11.472543 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:11.472645 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:11.472681 (+ 36us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0812 02:32:11.476685 (+ 4004us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36483/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0812 02:32:11.477889 (+ 1204us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.14.103.62:43501: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36483/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":356,"mutex_wait_us":871,"thread_start_us":114,"threads_started":1}
I20260812 02:32:11.479363 14748 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbep4M4/build/debug/bin/kudu with pid 20656
E20260812 02:32:11.491941 20767 webserver.cc:533] Webserver: sslize failed:
E20260812 02:32:11.492065 20767 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-08-12T02:32:11Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (379 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-08-12T02:32:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T02:32:11Z Disabled control of system clock
I20260812 02:32:11.530309 14748 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbep4M4/build/debug/bin/kudu
/tmp/dist-test-taskbep4M4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786501898713699-14748-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786501898713699-14748-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786501898713699-14748-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786501898713699-14748-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.103.62:43779
--webserver_interface=127.14.103.62
--webserver_port=0
--builtin_ntp_servers=127.14.103.20:44927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.103.62:43779
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786501898713699-14748-0/testchainca.cert with env {}
W20260812 02:32:11.659006 20771 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:32:11.659320 20771 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:32:11.659364 20771 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:32:11.664563 20771 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:32:11.664664 20771 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:32:11.664681 20771 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:32:11.664701 20771 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:32:11.664717 20771 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260812 02:32:11.664734 20771 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260812 02:32:11.664747 20771 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260812 02:32:11.671412 20771 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.103.20:44927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786501898713699-14748-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786501898713699-14748-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.103.62:43779
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.103.62:43779
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786501898713699-14748-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.103.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786501898713699-14748-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20771
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786501898713699-14748-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:32:11.672609 20771 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:32:11.673693 20771 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260812 02:32:11.683739 20771 server_base.cc:1034] running on GCE node
W20260812 02:32:11.683678 20777 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:32:11.683670 20776 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:32:11.683678 20779 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:32:11.684722 20771 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:32:11.685992 20771 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:32:11.689455 20771 hybrid_clock.cc:648] HybridClock initialized: now 1786501931689301 us; error 148 us; skew 500 ppm
I20260812 02:32:11.692528 20771 webserver.cc:460] Webserver started at http://127.14.103.62:40447/ using document root <none> and password file <none>
I20260812 02:32:11.693262 20771 fs_manager.cc:359] Metadata directory not provided
I20260812 02:32:11.693344 20771 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:32:11.693593 20771 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:32:11.696558 20771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786501898713699-14748-0/minicluster-data/master-0/data/instance:
uuid: "1fbae312b1204217ab0acb6f34ce163c"
format_stamp: "Formatted at 2026-08-12 02:32:11 on dist-test-slave-2w3w"
server_key: "c69106f8ad5ee671168ddc912e8d8d6c"
server_key_iv: "916d1bf3b4bd95c066de9cd9cafb28b7"
server_key_version: "encryptionkey@0"
I20260812 02:32:11.697307 20771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786501898713699-14748-0/minicluster-data/master-0/wal/instance:
uuid: "1fbae312b1204217ab0acb6f34ce163c"
format_stamp: "Formatted at 2026-08-12 02:32:11 on dist-test-slave-2w3w"
server_key: "c69106f8ad5ee671168ddc912e8d8d6c"
server_key_iv: "916d1bf3b4bd95c066de9cd9cafb28b7"
server_key_version: "encryptionkey@0"
I20260812 02:32:11.702700 20771 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260812 02:32:11.706636 20788 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:32:11.708734 20771 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 02:32:11.708976 20771 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786501898713699-14748-0/minicluster-data/master-0/data,/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786501898713699-14748-0/minicluster-data/master-0/wal
uuid: "1fbae312b1204217ab0acb6f34ce163c"
format_stamp: "Formatted at 2026-08-12 02:32:11 on dist-test-slave-2w3w"
server_key: "c69106f8ad5ee671168ddc912e8d8d6c"
server_key_iv: "916d1bf3b4bd95c066de9cd9cafb28b7"
server_key_version: "encryptionkey@0"
I20260812 02:32:11.709142 20771 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786501898713699-14748-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786501898713699-14748-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786501898713699-14748-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:32:11.722699 20771 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:32:11.723613 20771 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:32:11.723855 20771 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:32:11.732581 20771 rpc_server.cc:307] RPC server started. Bound to: 127.14.103.62:43779
I20260812 02:32:11.732600 20844 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.103.62:43779 every 8 connection(s)
I20260812 02:32:11.733708 20771 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786501898713699-14748-0/minicluster-data/master-0/data/info.pb
I20260812 02:32:11.736925 20845 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:32:11.740890 14748 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbep4M4/build/debug/bin/kudu as pid 20771
I20260812 02:32:11.741074 14748 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786501898713699-14748-0/minicluster-data/master-0/wal/instance
I20260812 02:32:11.741747 14748 external_mini_cluster.cc:1428] Setting key ecbb2cd28774cc5b3ca7f6bb04a7a746
I20260812 02:32:11.747435 20845 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1fbae312b1204217ab0acb6f34ce163c: Bootstrap starting.
I20260812 02:32:11.750201 20845 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1fbae312b1204217ab0acb6f34ce163c: Neither blocks nor log segments found. Creating new log.
I20260812 02:32:11.751479 20845 log.cc:824] T 00000000000000000000000000000000 P 1fbae312b1204217ab0acb6f34ce163c: Log is configured to *not* fsync() on all Append() calls
I20260812 02:32:11.754340 20845 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1fbae312b1204217ab0acb6f34ce163c: No bootstrap required, opened a new log
I20260812 02:32:11.754477 20848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:11.743796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43136 (local address 127.14.103.62:43779)
0812 02:32:11.744660 (+ 864us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:11.744678 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:11.744722 (+ 44us) server_negotiation.cc:407] Connection header received
0812 02:32:11.745484 (+ 762us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:11.745505 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:11.745993 (+ 488us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:11.746504 (+ 511us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:32:11.747222 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:11.748432 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:11.749641 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:11.750219 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:11.750569 (+ 350us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:11.751939 (+ 1370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:32:11.751946 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:32:11.751963 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:32:11.752626 (+ 663us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:32:11.752743 (+ 117us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:32:11.752779 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:32:11.752903 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0812 02:32:11.753174 (+ 271us) server_negotiation.cc:299] Negotiation successful
0812 02:32:11.753587 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":417,"thread_start_us":175,"threads_started":1}
I20260812 02:32:11.758535 14748 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 02:32:11.759618 20845 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1fbae312b1204217ab0acb6f34ce163c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fbae312b1204217ab0acb6f34ce163c" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 43779 } }
I20260812 02:32:11.759946 20845 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1fbae312b1204217ab0acb6f34ce163c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:32:11.760030 20845 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1fbae312b1204217ab0acb6f34ce163c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fbae312b1204217ab0acb6f34ce163c, State: Initialized, Role: FOLLOWER
I20260812 02:32:11.761106 20845 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1fbae312b1204217ab0acb6f34ce163c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fbae312b1204217ab0acb6f34ce163c" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 43779 } }
I20260812 02:32:11.761829 20845 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1fbae312b1204217ab0acb6f34ce163c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 02:32:11.761952 20845 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1fbae312b1204217ab0acb6f34ce163c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 02:32:11.762069 20845 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1fbae312b1204217ab0acb6f34ce163c [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:32:11.763329 20845 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1fbae312b1204217ab0acb6f34ce163c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fbae312b1204217ab0acb6f34ce163c" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 43779 } }
I20260812 02:32:11.763711 20845 leader_election.cc:302] T 00000000000000000000000000000000 P 1fbae312b1204217ab0acb6f34ce163c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1fbae312b1204217ab0acb6f34ce163c; no voters:
I20260812 02:32:11.764074 20845 leader_election.cc:288] T 00000000000000000000000000000000 P 1fbae312b1204217ab0acb6f34ce163c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 02:32:11.764298 20850 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1fbae312b1204217ab0acb6f34ce163c [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:32:11.764704 20850 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1fbae312b1204217ab0acb6f34ce163c [term 1 LEADER]: Becoming Leader. State: Replica: 1fbae312b1204217ab0acb6f34ce163c, State: Running, Role: LEADER
I20260812 02:32:11.765113 20850 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1fbae312b1204217ab0acb6f34ce163c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fbae312b1204217ab0acb6f34ce163c" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 43779 } }
I20260812 02:32:11.765578 20845 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1fbae312b1204217ab0acb6f34ce163c [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:32:11.767798 20852 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1fbae312b1204217ab0acb6f34ce163c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1fbae312b1204217ab0acb6f34ce163c. Latest consensus state: current_term: 1 leader_uuid: "1fbae312b1204217ab0acb6f34ce163c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fbae312b1204217ab0acb6f34ce163c" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 43779 } } }
I20260812 02:32:11.768333 20852 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1fbae312b1204217ab0acb6f34ce163c [sys.catalog]: This master's current role is: LEADER
I20260812 02:32:11.768790 20851 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1fbae312b1204217ab0acb6f34ce163c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1fbae312b1204217ab0acb6f34ce163c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fbae312b1204217ab0acb6f34ce163c" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 43779 } } }
I20260812 02:32:11.769325 20851 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1fbae312b1204217ab0acb6f34ce163c [sys.catalog]: This master's current role is: LEADER
W20260812 02:32:11.771595 20865 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1fbae312b1204217ab0acb6f34ce163c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 02:32:11.771708 20865 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 02:32:11.772061 20866 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 02:32:11.773389 20866 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 02:32:11.781847 20866 catalog_manager.cc:1347] Generated new cluster ID: 6c9b762d25c34b1891e40e11072fd4cc
I20260812 02:32:11.781953 20866 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 02:32:11.792685 20866 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 02:32:11.793977 20866 catalog_manager.cc:1504] Loading token signing keys...
I20260812 02:32:11.803457 20866 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1fbae312b1204217ab0acb6f34ce163c: Generated new TSK 0
I20260812 02:32:11.804337 20866 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 02:32:11.860074 20848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:11.820970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43148 (local address 127.14.103.62:43779)
0812 02:32:11.821142 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:11.821148 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:11.821414 (+ 266us) server_negotiation.cc:407] Connection header received
0812 02:32:11.821643 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:11.821647 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:11.821707 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:11.821814 (+ 107us) server_negotiation.cc:226] Negotiated authn=JWT
0812 02:32:11.822176 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:11.822796 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:11.824274 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:11.824607 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:11.824708 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:11.825484 (+ 776us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0812 02:32:11.859528 (+ 34044us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0812 02:32:11.859909 (+ 381us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:43148: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":66}
W20260812 02:32:11.860175 20848 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:43148: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260812 02:32:11.860594 20880 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:32:11.820864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:43779 (local address 127.0.0.1:43148)
0812 02:32:11.821258 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:11.821275 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 02:32:11.821428 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:32:11.821838 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:32:11.821841 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:32:11.821849 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0812 02:32:11.822019 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:11.822026 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:11.822928 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:11.822932 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:11.823989 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:11.824002 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:11.824768 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:11.824772 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:11.824888 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:11.824924 (+ 36us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0812 02:32:11.859927 (+ 35003us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0812 02:32:11.860389 (+ 462us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.14.103.62:43779: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":173,"threads_started":1}
I20260812 02:32:11.861310 14748 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbep4M4/build/debug/bin/kudu with pid 20771
2026-08-12T02:32:11Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (369 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (155 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (151 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501898713699-14748-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 02:32:12 dist-test-slave-2w3w krb5kdc[20893](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 02:32:12 dist-test-slave-2w3w krb5kdc[20893](info): set up 2 sockets
Aug 12 02:32:12 dist-test-slave-2w3w krb5kdc[20893](info): commencing operation
krb5kdc: starting...
W20260812 02:32:13.368954 14748 mini_kdc.cc:121] Time spent starting KDC: real 1.174s user 0.001s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501898713699-14748-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501898713699-14748-0/krb5kdc/test-user.keytab.
Aug 12 02:32:13 dist-test-slave-2w3w krb5kdc[20893](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501933, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T02:32:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T02:32:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.103.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.103.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.103.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.62.keytab.
Entry for principal kudu/127.14.103.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.62.keytab.
I20260812 02:32:13.527724 14748 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbep4M4/build/debug/bin/kudu
/tmp/dist-test-taskbep4M4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501898713699-14748-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501898713699-14748-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501898713699-14748-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501898713699-14748-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.103.62:36313
--webserver_interface=127.14.103.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.62.keytab
--principal=kudu/127.14.103.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.103.20:45061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.103.62:36313
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501898713699-14748-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501898713699-14748-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501898713699-14748-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:32:13.651521 20908 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:32:13.651901 20908 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:32:13.652005 20908 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:32:13.656543 20908 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:32:13.656675 20908 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:32:13.656710 20908 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:32:13.656754 20908 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:32:13.656788 20908 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 02:32:13.663354 20908 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.103.20:45061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501898713699-14748-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501898713699-14748-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.103.62:36313
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.62.keytab
--principal=kudu/127.14.103.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.103.62:36313
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501898713699-14748-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.103.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20908
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501898713699-14748-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:32:13.664638 20908 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:32:13.665810 20908 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:32:13.675711 20914 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:32:13.675748 20917 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:32:13.675833 20908 server_base.cc:1034] running on GCE node
W20260812 02:32:13.676017 20915 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:32:13.676914 20908 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:32:13.678179 20908 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:32:13.679401 20908 hybrid_clock.cc:648] HybridClock initialized: now 1786501933679372 us; error 50 us; skew 500 ppm
Aug 12 02:32:13 dist-test-slave-2w3w krb5kdc[20893](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501933, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:32:13.682780 20908 init.cc:377] Logged in from keytab as kudu/127.14.103.62@KRBTEST.COM (short username kudu)
I20260812 02:32:13.683868 20908 webserver.cc:460] Webserver started at http://127.14.103.62:37405/ using document root <none> and password file <none>
I20260812 02:32:13.684509 20908 fs_manager.cc:359] Metadata directory not provided
I20260812 02:32:13.684612 20908 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:32:13.684857 20908 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:32:13.687207 20908 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501898713699-14748-0/minicluster-data/master-0/data/instance:
uuid: "1c0840b7fa964d1eb069aa6af32139a5"
format_stamp: "Formatted at 2026-08-12 02:32:13 on dist-test-slave-2w3w"
server_key: "0051dc5a5f960c1e20ee044f48c965fb"
server_key_iv: "f3fc39838ff7ae11dc9932288ef509c1"
server_key_version: "encryptionkey@0"
I20260812 02:32:13.687856 20908 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501898713699-14748-0/minicluster-data/master-0/wal/instance:
uuid: "1c0840b7fa964d1eb069aa6af32139a5"
format_stamp: "Formatted at 2026-08-12 02:32:13 on dist-test-slave-2w3w"
server_key: "0051dc5a5f960c1e20ee044f48c965fb"
server_key_iv: "f3fc39838ff7ae11dc9932288ef509c1"
server_key_version: "encryptionkey@0"
I20260812 02:32:13.692230 20908 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260812 02:32:13.695272 20924 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:32:13.696970 20908 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260812 02:32:13.697165 20908 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501898713699-14748-0/minicluster-data/master-0/data,/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501898713699-14748-0/minicluster-data/master-0/wal
uuid: "1c0840b7fa964d1eb069aa6af32139a5"
format_stamp: "Formatted at 2026-08-12 02:32:13 on dist-test-slave-2w3w"
server_key: "0051dc5a5f960c1e20ee044f48c965fb"
server_key_iv: "f3fc39838ff7ae11dc9932288ef509c1"
server_key_version: "encryptionkey@0"
I20260812 02:32:13.697320 20908 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501898713699-14748-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501898713699-14748-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501898713699-14748-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:32:13.748260 20908 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:32:13.752192 20908 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:32:13.752465 20908 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:32:13.761330 20908 rpc_server.cc:307] RPC server started. Bound to: 127.14.103.62:36313
I20260812 02:32:13.761356 20980 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.103.62:36313 every 8 connection(s)
I20260812 02:32:13.762410 20908 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501898713699-14748-0/minicluster-data/master-0/data/info.pb
I20260812 02:32:13.765527 20981 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:32:13.767931 14748 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbep4M4/build/debug/bin/kudu as pid 20908
I20260812 02:32:13.768074 14748 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501898713699-14748-0/minicluster-data/master-0/wal/instance
I20260812 02:32:13.768404 14748 external_mini_cluster.cc:1428] Setting key 2a7bf67075bc26340ac42e6562e34fd1
I20260812 02:32:13.773265 20981 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c0840b7fa964d1eb069aa6af32139a5: Bootstrap starting.
I20260812 02:32:13.775913 20981 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1c0840b7fa964d1eb069aa6af32139a5: Neither blocks nor log segments found. Creating new log.
I20260812 02:32:13.776727 20981 log.cc:824] T 00000000000000000000000000000000 P 1c0840b7fa964d1eb069aa6af32139a5: Log is configured to *not* fsync() on all Append() calls
Aug 12 02:32:13 dist-test-slave-2w3w krb5kdc[20893](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501933, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.103.62@KRBTEST.COM
I20260812 02:32:13.780652 20981 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c0840b7fa964d1eb069aa6af32139a5: No bootstrap required, opened a new log
I20260812 02:32:13.785454 20981 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1c0840b7fa964d1eb069aa6af32139a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c0840b7fa964d1eb069aa6af32139a5" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 36313 } }
I20260812 02:32:13.785974 20981 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1c0840b7fa964d1eb069aa6af32139a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:32:13.786392 20981 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1c0840b7fa964d1eb069aa6af32139a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c0840b7fa964d1eb069aa6af32139a5, State: Initialized, Role: FOLLOWER
I20260812 02:32:13.786607 20985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:13.770003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52214 (local address 127.14.103.62:36313)
0812 02:32:13.770557 (+ 554us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:13.770574 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:13.770614 (+ 40us) server_negotiation.cc:407] Connection header received
0812 02:32:13.771478 (+ 864us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:13.771500 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:13.771885 (+ 385us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:13.772260 (+ 375us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:32:13.773176 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:13.774257 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:13.775263 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:13.775783 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:13.775971 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:13.779570 (+ 3599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:32:13.779577 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:32:13.779593 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:32:13.779625 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:32:13.782040 (+ 2415us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:13.782624 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:13.782630 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:13.782635 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:13.782759 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:13.783161 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:13.783165 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:13.783167 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:13.784018 (+ 851us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:32:13.784454 (+ 436us) server_negotiation.cc:1035] Waiting for connection context
0812 02:32:13.785005 (+ 551us) server_negotiation.cc:299] Negotiation successful
0812 02:32:13.785539 (+ 534us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":169,"threads_started":1}
I20260812 02:32:13.787727 20981 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1c0840b7fa964d1eb069aa6af32139a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c0840b7fa964d1eb069aa6af32139a5" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 36313 } }
I20260812 02:32:13.788048 20981 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1c0840b7fa964d1eb069aa6af32139a5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 02:32:13.788125 20981 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1c0840b7fa964d1eb069aa6af32139a5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 02:32:13.788234 20981 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1c0840b7fa964d1eb069aa6af32139a5 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:32:13.789747 20981 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1c0840b7fa964d1eb069aa6af32139a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c0840b7fa964d1eb069aa6af32139a5" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 36313 } }
I20260812 02:32:13.790179 20981 leader_election.cc:302] T 00000000000000000000000000000000 P 1c0840b7fa964d1eb069aa6af32139a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1c0840b7fa964d1eb069aa6af32139a5; no voters:
I20260812 02:32:13.790539 20981 leader_election.cc:288] T 00000000000000000000000000000000 P 1c0840b7fa964d1eb069aa6af32139a5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 02:32:13.791414 20987 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1c0840b7fa964d1eb069aa6af32139a5 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:32:13.791700 20987 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1c0840b7fa964d1eb069aa6af32139a5 [term 1 LEADER]: Becoming Leader. State: Replica: 1c0840b7fa964d1eb069aa6af32139a5, State: Running, Role: LEADER
I20260812 02:32:13.792119 20987 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1c0840b7fa964d1eb069aa6af32139a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c0840b7fa964d1eb069aa6af32139a5" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 36313 } }
I20260812 02:32:13.792165 20981 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1c0840b7fa964d1eb069aa6af32139a5 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:32:13.794183 20988 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c0840b7fa964d1eb069aa6af32139a5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1c0840b7fa964d1eb069aa6af32139a5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c0840b7fa964d1eb069aa6af32139a5" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 36313 } } }
I20260812 02:32:13.794291 20990 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c0840b7fa964d1eb069aa6af32139a5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1c0840b7fa964d1eb069aa6af32139a5. Latest consensus state: current_term: 1 leader_uuid: "1c0840b7fa964d1eb069aa6af32139a5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c0840b7fa964d1eb069aa6af32139a5" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 36313 } } }
I20260812 02:32:13.794584 20990 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c0840b7fa964d1eb069aa6af32139a5 [sys.catalog]: This master's current role is: LEADER
I20260812 02:32:13.794584 20988 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c0840b7fa964d1eb069aa6af32139a5 [sys.catalog]: This master's current role is: LEADER
W20260812 02:32:13.798724 21002 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1c0840b7fa964d1eb069aa6af32139a5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 02:32:13.798918 21002 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 02:32:13.799041 21003 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 02:32:13.800038 21003 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 02:32:13.808852 21003 catalog_manager.cc:1347] Generated new cluster ID: ff248d09395b428ba508e2a9044de9d6
I20260812 02:32:13.808977 21003 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 02:32:13.823866 21003 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 02:32:13.825134 21003 catalog_manager.cc:1504] Loading token signing keys...
I20260812 02:32:13.837486 21003 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1c0840b7fa964d1eb069aa6af32139a5: Generated new TSK 0
I20260812 02:32:13.838490 21003 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.103.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.103.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.103.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.1.keytab.
Entry for principal kudu/127.14.103.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.1.keytab.
I20260812 02:32:13.888149 14748 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbep4M4/build/debug/bin/kudu
/tmp/dist-test-taskbep4M4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501898713699-14748-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501898713699-14748-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501898713699-14748-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501898713699-14748-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.103.1:0
--local_ip_for_outbound_sockets=127.14.103.1
--webserver_interface=127.14.103.1
--webserver_port=0
--tserver_master_addrs=127.14.103.62:36313
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.1.keytab
--principal=kudu/127.14.103.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.103.20:45061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501898713699-14748-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501898713699-14748-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501898713699-14748-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:32:14.027793 21010 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:32:14.028143 21010 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:32:14.028193 21010 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:32:14.032737 21010 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:32:14.032831 21010 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:32:14.032905 21010 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.103.1
I20260812 02:32:14.041289 21010 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.103.20:45061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501898713699-14748-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501898713699-14748-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.1.keytab
--principal=kudu/127.14.103.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.103.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501898713699-14748-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.103.1
--webserver_port=0
--tserver_master_addrs=127.14.103.62:36313
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21010
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.103.1
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501898713699-14748-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:32:14.042572 21010 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:32:14.043776 21010 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260812 02:32:14.055033 21010 server_base.cc:1034] running on GCE node
W20260812 02:32:14.054906 21015 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:32:14.055069 21017 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:32:14.054906 21019 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:32:14.055819 21010 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:32:14.056491 21010 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:32:14.057675 21010 hybrid_clock.cc:648] HybridClock initialized: now 1786501934057656 us; error 34 us; skew 500 ppm
Aug 12 02:32:14 dist-test-slave-2w3w krb5kdc[20893](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501934, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:32:14.061271 21010 init.cc:377] Logged in from keytab as kudu/127.14.103.1@KRBTEST.COM (short username kudu)
I20260812 02:32:14.062430 21010 webserver.cc:460] Webserver started at http://127.14.103.1:33761/ using document root <none> and password file <none>
I20260812 02:32:14.063139 21010 fs_manager.cc:359] Metadata directory not provided
I20260812 02:32:14.063249 21010 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:32:14.063511 21010 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:32:14.066044 21010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501898713699-14748-0/minicluster-data/ts-0/data/instance:
uuid: "aad429286bb8413185fbab2587668504"
format_stamp: "Formatted at 2026-08-12 02:32:14 on dist-test-slave-2w3w"
server_key: "278728ca97a03c150013c51c3658671e"
server_key_iv: "a1bdcb8f478d93be23cb053d87e5653a"
server_key_version: "encryptionkey@0"
I20260812 02:32:14.066826 21010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501898713699-14748-0/minicluster-data/ts-0/wal/instance:
uuid: "aad429286bb8413185fbab2587668504"
format_stamp: "Formatted at 2026-08-12 02:32:14 on dist-test-slave-2w3w"
server_key: "278728ca97a03c150013c51c3658671e"
server_key_iv: "a1bdcb8f478d93be23cb053d87e5653a"
server_key_version: "encryptionkey@0"
I20260812 02:32:14.071584 21010 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260812 02:32:14.075065 21028 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:32:14.076581 21010 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 02:32:14.076728 21010 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501898713699-14748-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501898713699-14748-0/minicluster-data/ts-0/wal
uuid: "aad429286bb8413185fbab2587668504"
format_stamp: "Formatted at 2026-08-12 02:32:14 on dist-test-slave-2w3w"
server_key: "278728ca97a03c150013c51c3658671e"
server_key_iv: "a1bdcb8f478d93be23cb053d87e5653a"
server_key_version: "encryptionkey@0"
I20260812 02:32:14.076903 21010 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501898713699-14748-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501898713699-14748-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501898713699-14748-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:32:14.110529 21010 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:32:14.114820 21010 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:32:14.115053 21010 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:32:14.115836 21010 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:32:14.116784 21010 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:32:14.116842 21010 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:32:14.116886 21010 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:32:14.116901 21010 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:32:14.127278 21010 rpc_server.cc:307] RPC server started. Bound to: 127.14.103.1:36451
I20260812 02:32:14.127300 21148 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.103.1:36451 every 8 connection(s)
I20260812 02:32:14.128306 21010 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501898713699-14748-0/minicluster-data/ts-0/data/info.pb
I20260812 02:32:14.128824 14748 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbep4M4/build/debug/bin/kudu as pid 21010
I20260812 02:32:14.128985 14748 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501898713699-14748-0/minicluster-data/ts-0/wal/instance
I20260812 02:32:14.129400 14748 external_mini_cluster.cc:1428] Setting key 0dad02e0bd8a163f2a39ef361c724d34
WARNING: no policy specified for kudu/127.14.103.2@KRBTEST.COM; defaulting to no policy
Aug 12 02:32:14 dist-test-slave-2w3w krb5kdc[20893](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501934, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.1@KRBTEST.COM for kudu/127.14.103.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.103.2@KRBTEST.COM" created.
I20260812 02:32:14.148427 20985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:14.130679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.1:50939 (local address 127.14.103.62:36313)
0812 02:32:14.130872 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:14.130879 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:14.131985 (+ 1106us) server_negotiation.cc:407] Connection header received
0812 02:32:14.133690 (+ 1705us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:14.133697 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:14.133750 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:14.133843 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:32:14.135768 (+ 1925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:14.136385 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:14.138557 (+ 2172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:14.138814 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:14.138932 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:14.142689 (+ 3757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:32:14.142693 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:32:14.142695 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:32:14.142729 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:32:14.145010 (+ 2281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:14.145912 (+ 902us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:14.145916 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:14.145917 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:14.146018 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:14.146652 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:14.146655 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:14.146656 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:14.146905 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:32:14.147200 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0812 02:32:14.147999 (+ 799us) server_negotiation.cc:299] Negotiation successful
0812 02:32:14.148186 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260812 02:32:14.150003 21151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:14.131260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:36313 (local address 127.14.103.1:50939)
0812 02:32:14.131788 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:14.131847 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0812 02:32:14.133378 (+ 1531us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:32:14.134072 (+ 694us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:32:14.134083 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:32:14.134723 (+ 640us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:32:14.135559 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:14.135584 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:14.136552 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:14.136557 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:14.138295 (+ 1738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:14.138314 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:14.139074 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:14.139080 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:14.139335 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:14.140090 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:32:14.140115 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:32:14.142321 (+ 2206us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:32:14.145247 (+ 2926us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:32:14.145255 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:32:14.145274 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:32:14.145644 (+ 370us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:32:14.146158 (+ 514us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:32:14.146161 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:32:14.146163 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:32:14.146491 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:32:14.147132 (+ 641us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:32:14.147138 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:32:14.147573 (+ 435us) client_negotiation.cc:769] Sending connection context
0812 02:32:14.147761 (+ 188us) client_negotiation.cc:240] Negotiation successful
0812 02:32:14.148176 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":176,"threads_started":1}
I20260812 02:32:14.152459 21149 heartbeater.cc:347] Connected to a master server at 127.14.103.62:36313
I20260812 02:32:14.152796 21149 heartbeater.cc:464] Registering TS with master...
I20260812 02:32:14.153401 21149 heartbeater.cc:511] Master 127.14.103.62:36313 requested a full tablet report, sending...
I20260812 02:32:14.155364 20943 ts_manager.cc:194] Registered new tserver with Master: aad429286bb8413185fbab2587668504 (127.14.103.1:36451)
I20260812 02:32:14.156595 20943 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.103.1@KRBTEST.COM'} at 127.14.103.1:50939
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.103.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.2.keytab.
Entry for principal kudu/127.14.103.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.2.keytab.
I20260812 02:32:14.172115 14748 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbep4M4/build/debug/bin/kudu
/tmp/dist-test-taskbep4M4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501898713699-14748-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501898713699-14748-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501898713699-14748-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501898713699-14748-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.103.2:0
--local_ip_for_outbound_sockets=127.14.103.2
--webserver_interface=127.14.103.2
--webserver_port=0
--tserver_master_addrs=127.14.103.62:36313
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.2.keytab
--principal=kudu/127.14.103.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.103.20:45061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501898713699-14748-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501898713699-14748-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501898713699-14748-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:32:14.298026 21155 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:32:14.298363 21155 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:32:14.298413 21155 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:32:14.303435 21155 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:32:14.303530 21155 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:32:14.303658 21155 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.103.2
I20260812 02:32:14.311571 21155 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.103.20:45061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501898713699-14748-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501898713699-14748-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.2.keytab
--principal=kudu/127.14.103.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.103.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501898713699-14748-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.103.2
--webserver_port=0
--tserver_master_addrs=127.14.103.62:36313
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21155
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.103.2
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501898713699-14748-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:32:14.312922 21155 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:32:14.314522 21155 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:32:14.326076 21160 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:32:14.326114 21161 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:32:14.326076 21163 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:32:14.326396 21155 server_base.cc:1034] running on GCE node
I20260812 02:32:14.327144 21155 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:32:14.327910 21155 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:32:14.329190 21155 hybrid_clock.cc:648] HybridClock initialized: now 1786501934329177 us; error 31 us; skew 500 ppm
Aug 12 02:32:14 dist-test-slave-2w3w krb5kdc[20893](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501934, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:32:14.332707 21155 init.cc:377] Logged in from keytab as kudu/127.14.103.2@KRBTEST.COM (short username kudu)
I20260812 02:32:14.333691 21155 webserver.cc:460] Webserver started at http://127.14.103.2:41115/ using document root <none> and password file <none>
I20260812 02:32:14.334261 21155 fs_manager.cc:359] Metadata directory not provided
I20260812 02:32:14.334328 21155 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:32:14.334524 21155 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:32:14.336922 21155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501898713699-14748-0/minicluster-data/ts-1/data/instance:
uuid: "f5bf95c406e74bdebeca729575d7f967"
format_stamp: "Formatted at 2026-08-12 02:32:14 on dist-test-slave-2w3w"
server_key: "e10952331d65784ada3c53ca2917ce33"
server_key_iv: "0c5ffa92c90a12c2a516a0ca62d283fd"
server_key_version: "encryptionkey@0"
I20260812 02:32:14.337594 21155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501898713699-14748-0/minicluster-data/ts-1/wal/instance:
uuid: "f5bf95c406e74bdebeca729575d7f967"
format_stamp: "Formatted at 2026-08-12 02:32:14 on dist-test-slave-2w3w"
server_key: "e10952331d65784ada3c53ca2917ce33"
server_key_iv: "0c5ffa92c90a12c2a516a0ca62d283fd"
server_key_version: "encryptionkey@0"
I20260812 02:32:14.342098 21155 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260812 02:32:14.345328 21172 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:32:14.347074 21155 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260812 02:32:14.347265 21155 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501898713699-14748-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501898713699-14748-0/minicluster-data/ts-1/wal
uuid: "f5bf95c406e74bdebeca729575d7f967"
format_stamp: "Formatted at 2026-08-12 02:32:14 on dist-test-slave-2w3w"
server_key: "e10952331d65784ada3c53ca2917ce33"
server_key_iv: "0c5ffa92c90a12c2a516a0ca62d283fd"
server_key_version: "encryptionkey@0"
I20260812 02:32:14.347420 21155 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501898713699-14748-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501898713699-14748-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501898713699-14748-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:32:14.383545 21155 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:32:14.387501 21155 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:32:14.387746 21155 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:32:14.388458 21155 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:32:14.389377 21155 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:32:14.389428 21155 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:32:14.389473 21155 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:32:14.389530 21155 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:32:14.400375 21155 rpc_server.cc:307] RPC server started. Bound to: 127.14.103.2:37767
I20260812 02:32:14.400393 21297 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.103.2:37767 every 8 connection(s)
I20260812 02:32:14.401405 21155 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501898713699-14748-0/minicluster-data/ts-1/data/info.pb
I20260812 02:32:14.402508 14748 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbep4M4/build/debug/bin/kudu as pid 21155
I20260812 02:32:14.402652 14748 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501898713699-14748-0/minicluster-data/ts-1/wal/instance
I20260812 02:32:14.403021 14748 external_mini_cluster.cc:1428] Setting key cb237819374f5260f01679e0033de419
Aug 12 02:32:14 dist-test-slave-2w3w krb5kdc[20893](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501934, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.2@KRBTEST.COM for kudu/127.14.103.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.14.103.3@KRBTEST.COM; defaulting to no policy
I20260812 02:32:14.419787 20985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:14.403750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.2:45661 (local address 127.14.103.62:36313)
0812 02:32:14.403936 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:14.403943 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:14.405024 (+ 1081us) server_negotiation.cc:407] Connection header received
0812 02:32:14.406446 (+ 1422us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:14.406449 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:14.406512 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:14.406596 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:32:14.407964 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:14.408522 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:14.410930 (+ 2408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:14.411370 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:14.411519 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:14.415272 (+ 3753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:32:14.415275 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:32:14.415283 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:32:14.415314 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:32:14.417043 (+ 1729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:14.417938 (+ 895us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:14.417942 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:14.417943 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:14.418039 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:14.418440 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:14.418442 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:14.418443 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:14.418653 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:32:14.418746 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0812 02:32:14.419503 (+ 757us) server_negotiation.cc:299] Negotiation successful
0812 02:32:14.419645 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.103.3@KRBTEST.COM" created.
I20260812 02:32:14.421092 21300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:14.404327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:36313 (local address 127.14.103.2:45661)
0812 02:32:14.404877 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:14.404925 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0812 02:32:14.406211 (+ 1286us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:32:14.406794 (+ 583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:32:14.406801 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:32:14.407269 (+ 468us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:32:14.407804 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:14.407819 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:14.408668 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:14.408674 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:14.410685 (+ 2011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:14.410709 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:14.411633 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:14.411641 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:14.411934 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:14.412755 (+ 821us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:32:14.412789 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:32:14.414952 (+ 2163us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:32:14.417248 (+ 2296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:32:14.417254 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:32:14.417272 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:32:14.417787 (+ 515us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:32:14.418148 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:32:14.418151 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:32:14.418152 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:32:14.418342 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:32:14.418791 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:32:14.418798 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:32:14.419201 (+ 403us) client_negotiation.cc:769] Sending connection context
0812 02:32:14.419457 (+ 256us) client_negotiation.cc:240] Negotiation successful
0812 02:32:14.419816 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":139,"threads_started":1}
I20260812 02:32:14.423190 21298 heartbeater.cc:347] Connected to a master server at 127.14.103.62:36313
I20260812 02:32:14.423549 21298 heartbeater.cc:464] Registering TS with master...
I20260812 02:32:14.424284 21298 heartbeater.cc:511] Master 127.14.103.62:36313 requested a full tablet report, sending...
I20260812 02:32:14.425849 20943 ts_manager.cc:194] Registered new tserver with Master: f5bf95c406e74bdebeca729575d7f967 (127.14.103.2:37767)
I20260812 02:32:14.426733 20943 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.103.2@KRBTEST.COM'} at 127.14.103.2:45661
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.103.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.3.keytab.
Entry for principal kudu/127.14.103.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.3.keytab.
I20260812 02:32:14.443675 14748 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbep4M4/build/debug/bin/kudu
/tmp/dist-test-taskbep4M4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501898713699-14748-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501898713699-14748-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501898713699-14748-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501898713699-14748-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.103.3:0
--local_ip_for_outbound_sockets=127.14.103.3
--webserver_interface=127.14.103.3
--webserver_port=0
--tserver_master_addrs=127.14.103.62:36313
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.3.keytab
--principal=kudu/127.14.103.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.103.20:45061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501898713699-14748-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501898713699-14748-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501898713699-14748-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:32:14.569257 21303 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:32:14.569644 21303 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:32:14.569762 21303 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:32:14.574393 21303 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:32:14.574533 21303 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:32:14.574632 21303 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.103.3
I20260812 02:32:14.581027 21303 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.103.20:45061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501898713699-14748-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501898713699-14748-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.3.keytab
--principal=kudu/127.14.103.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.103.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501898713699-14748-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.103.3
--webserver_port=0
--tserver_master_addrs=127.14.103.62:36313
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21303
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.103.3
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501898713699-14748-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:32:14.582391 21303 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:32:14.583606 21303 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:32:14.595357 21309 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:32:14.595520 21308 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:32:14.595585 21303 server_base.cc:1034] running on GCE node
W20260812 02:32:14.595396 21311 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:32:14.596225 21303 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:32:14.597029 21303 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:32:14.598248 21303 hybrid_clock.cc:648] HybridClock initialized: now 1786501934598221 us; error 45 us; skew 500 ppm
Aug 12 02:32:14 dist-test-slave-2w3w krb5kdc[20893](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501934, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:32:14.601881 21303 init.cc:377] Logged in from keytab as kudu/127.14.103.3@KRBTEST.COM (short username kudu)
I20260812 02:32:14.603243 21303 webserver.cc:460] Webserver started at http://127.14.103.3:46203/ using document root <none> and password file <none>
I20260812 02:32:14.603890 21303 fs_manager.cc:359] Metadata directory not provided
I20260812 02:32:14.603962 21303 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:32:14.604195 21303 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:32:14.606710 21303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501898713699-14748-0/minicluster-data/ts-2/data/instance:
uuid: "203031bf259142c2a22e8c8f398aa47f"
format_stamp: "Formatted at 2026-08-12 02:32:14 on dist-test-slave-2w3w"
server_key: "6c6c055dce89e32f7de57bf6cc287085"
server_key_iv: "018f2332a8177110958a42057bdabfb2"
server_key_version: "encryptionkey@0"
I20260812 02:32:14.607501 21303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501898713699-14748-0/minicluster-data/ts-2/wal/instance:
uuid: "203031bf259142c2a22e8c8f398aa47f"
format_stamp: "Formatted at 2026-08-12 02:32:14 on dist-test-slave-2w3w"
server_key: "6c6c055dce89e32f7de57bf6cc287085"
server_key_iv: "018f2332a8177110958a42057bdabfb2"
server_key_version: "encryptionkey@0"
I20260812 02:32:14.612861 21303 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260812 02:32:14.616250 21320 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:32:14.617795 21303 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260812 02:32:14.617977 21303 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501898713699-14748-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501898713699-14748-0/minicluster-data/ts-2/wal
uuid: "203031bf259142c2a22e8c8f398aa47f"
format_stamp: "Formatted at 2026-08-12 02:32:14 on dist-test-slave-2w3w"
server_key: "6c6c055dce89e32f7de57bf6cc287085"
server_key_iv: "018f2332a8177110958a42057bdabfb2"
server_key_version: "encryptionkey@0"
I20260812 02:32:14.618163 21303 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501898713699-14748-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501898713699-14748-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501898713699-14748-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:32:14.641058 21303 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:32:14.645146 21303 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:32:14.645406 21303 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:32:14.646082 21303 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:32:14.647022 21303 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:32:14.647070 21303 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:32:14.647140 21303 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:32:14.647172 21303 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:32:14.657183 21303 rpc_server.cc:307] RPC server started. Bound to: 127.14.103.3:38943
I20260812 02:32:14.657205 21446 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.103.3:38943 every 8 connection(s)
I20260812 02:32:14.658161 21303 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501898713699-14748-0/minicluster-data/ts-2/data/info.pb
I20260812 02:32:14.663255 14748 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbep4M4/build/debug/bin/kudu as pid 21303
I20260812 02:32:14.663381 14748 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501898713699-14748-0/minicluster-data/ts-2/wal/instance
I20260812 02:32:14.663694 14748 external_mini_cluster.cc:1428] Setting key 46462f77e4a3c90557cf51dce6025aaf
Aug 12 02:32:14 dist-test-slave-2w3w krb5kdc[20893](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501934, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.3@KRBTEST.COM for kudu/127.14.103.62@KRBTEST.COM
I20260812 02:32:14.675859 20985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:14.660366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.3:33555 (local address 127.14.103.62:36313)
0812 02:32:14.660544 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:14.660548 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:14.661497 (+ 949us) server_negotiation.cc:407] Connection header received
0812 02:32:14.663107 (+ 1610us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:14.663111 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:14.663180 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:14.663270 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:32:14.664664 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:14.665228 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:14.666502 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:14.666744 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:14.666872 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:14.670674 (+ 3802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:32:14.670678 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:32:14.670680 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:32:14.670722 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:32:14.672845 (+ 2123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:14.673778 (+ 933us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:14.673782 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:14.673784 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:14.674000 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:14.674537 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:14.674540 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:14.674541 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:14.674768 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:32:14.674894 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0812 02:32:14.675533 (+ 639us) server_negotiation.cc:299] Negotiation successful
0812 02:32:14.675686 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260812 02:32:14.676853 21449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:14.660752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:36313 (local address 127.14.103.3:33555)
0812 02:32:14.661321 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:14.661370 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0812 02:32:14.662827 (+ 1457us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:32:14.663488 (+ 661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:32:14.663497 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:32:14.663965 (+ 468us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:32:14.664506 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:14.664520 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:14.665376 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:14.665380 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:14.666349 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:14.666360 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:14.666939 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:14.666944 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:14.667163 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:14.667852 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:32:14.667875 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:32:14.670358 (+ 2483us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:32:14.673099 (+ 2741us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:32:14.673107 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:32:14.673150 (+ 43us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:32:14.673636 (+ 486us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:32:14.674171 (+ 535us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:32:14.674175 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:32:14.674177 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:32:14.674384 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:32:14.674893 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:32:14.674903 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:32:14.675264 (+ 361us) client_negotiation.cc:769] Sending connection context
0812 02:32:14.675480 (+ 216us) client_negotiation.cc:240] Negotiation successful
0812 02:32:14.675860 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":135,"threads_started":1}
I20260812 02:32:14.678792 21447 heartbeater.cc:347] Connected to a master server at 127.14.103.62:36313
I20260812 02:32:14.679159 21447 heartbeater.cc:464] Registering TS with master...
I20260812 02:32:14.679929 21447 heartbeater.cc:511] Master 127.14.103.62:36313 requested a full tablet report, sending...
I20260812 02:32:14.681272 20943 ts_manager.cc:194] Registered new tserver with Master: 203031bf259142c2a22e8c8f398aa47f (127.14.103.3:38943)
I20260812 02:32:14.681998 20943 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.103.3@KRBTEST.COM'} at 127.14.103.3:33555
I20260812 02:32:14.692593 14748 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501898713699-14748-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501898713699-14748-0/krb5kdc/test-admin.keytab.
Aug 12 02:32:14 dist-test-slave-2w3w krb5kdc[20893](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501934, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:32:14.718489 14748 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260812 02:32:14.722148 21454 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:32:14.720457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60638 (local address 127.14.103.1:36451)
0812 02:32:14.720810 (+ 353us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:14.720816 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:14.720842 (+ 26us) server_negotiation.cc:407] Connection header received
0812 02:32:14.720903 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:14.720910 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:14.721228 (+ 318us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:14.721387 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:32:14.721968 (+ 581us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60638: BlockingRecv error: recv got EOF from 127.0.0.1:60638 (error 108)
Metrics: {"server-negotiator.queue_time_us":239,"thread_start_us":119,"threads_started":1}
W20260812 02:32:14.722654 14748 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501898713699-14748-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501898713699-14748-0/krb5kdc/krb5cc)
Aug 12 02:32:14 dist-test-slave-2w3w krb5kdc[20893](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501934, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:32:14.723999 14748 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 12 02:32:14 dist-test-slave-2w3w krb5kdc[20893](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501934, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:32:14.725340 14748 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 12 02:32:14 dist-test-slave-2w3w krb5kdc[20893](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501934, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.103.1@KRBTEST.COM
I20260812 02:32:14.737609 21454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:14.726558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60654 (local address 127.14.103.1:36451)
0812 02:32:14.726691 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:14.726695 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:14.726726 (+ 31us) server_negotiation.cc:407] Connection header received
0812 02:32:14.726957 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:14.727039 (+ 82us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:14.727109 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:14.727190 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:32:14.727809 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:14.728610 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:14.729596 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:14.730231 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:14.730340 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:14.733154 (+ 2814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:32:14.733169 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:32:14.733184 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:32:14.733217 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:32:14.735415 (+ 2198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:14.735963 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:14.735969 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:14.735971 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:14.736068 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:14.736491 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:14.736495 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:14.736497 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:14.736814 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:32:14.737022 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0812 02:32:14.737293 (+ 271us) server_negotiation.cc:299] Negotiation successful
0812 02:32:14.737471 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
W20260812 02:32:14.742838 21454 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:32:14.741462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60668 (local address 127.14.103.1:36451)
0812 02:32:14.741628 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:14.741631 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:14.741645 (+ 14us) server_negotiation.cc:407] Connection header received
0812 02:32:14.741792 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:14.741794 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:14.741839 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:14.741912 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:32:14.742721 (+ 809us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60668: BlockingRecv error: recv got EOF from 127.0.0.1:60668 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
W20260812 02:32:14.743019 14748 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501898713699-14748-0/krb5kdc/krb5cc' of type FILE
Aug 12 02:32:14 dist-test-slave-2w3w krb5kdc[20893](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501934, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:32:14.744210 14748 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 12 02:32:14 dist-test-slave-2w3w krb5kdc[20893](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501934, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:32:14.745221 14748 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 12 02:32:14 dist-test-slave-2w3w krb5kdc[20893](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501934, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.103.1@KRBTEST.COM
I20260812 02:32:14.755685 21454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:14.746297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60682 (local address 127.14.103.1:36451)
0812 02:32:14.746511 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:14.746518 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:14.746581 (+ 63us) server_negotiation.cc:407] Connection header received
0812 02:32:14.746752 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:14.746756 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:14.746809 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:14.746903 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:32:14.747576 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:14.748120 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:14.748990 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:14.749252 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:14.749377 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:14.752167 (+ 2790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:32:14.752170 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:32:14.752172 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:32:14.752204 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:32:14.753835 (+ 1631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:14.754350 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:14.754354 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:14.754355 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:14.754447 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:14.754737 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:14.754755 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:14.754757 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:14.754954 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:32:14.755171 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0812 02:32:14.755399 (+ 228us) server_negotiation.cc:299] Negotiation successful
0812 02:32:14.755522 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
W20260812 02:32:14.759088 21454 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:32:14.757914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60688 (local address 127.14.103.1:36451)
0812 02:32:14.758057 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:14.758061 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:14.758180 (+ 119us) server_negotiation.cc:407] Connection header received
0812 02:32:14.758332 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:14.758335 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:14.758415 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:14.758505 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:32:14.758938 (+ 433us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60688: BlockingRecv error: recv got EOF from 127.0.0.1:60688 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260812 02:32:14.759375 14748 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786501898713699-14748-0/krb5kdc/krb5cc' of type FILE
Aug 12 02:32:14 dist-test-slave-2w3w krb5kdc[20893](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501934, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:32:14.760592 14748 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 12 02:32:14 dist-test-slave-2w3w krb5kdc[20893](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501934, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:32:14.761906 14748 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 12 02:32:14 dist-test-slave-2w3w krb5kdc[20893](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501934, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.103.1@KRBTEST.COM
I20260812 02:32:14.772560 21454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:14.763358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60694 (local address 127.14.103.1:36451)
0812 02:32:14.763642 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:14.763646 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:14.763663 (+ 17us) server_negotiation.cc:407] Connection header received
0812 02:32:14.763833 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:14.763836 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:14.763888 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:14.763978 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:32:14.764598 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:14.765125 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:14.765979 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:14.766211 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:14.766323 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:14.769164 (+ 2841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:32:14.769168 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:32:14.769170 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:32:14.769201 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:32:14.770693 (+ 1492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:14.771280 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:14.771288 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:14.771290 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:14.771371 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:14.771656 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:14.771660 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:14.771662 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:14.771856 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:32:14.772027 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0812 02:32:14.772256 (+ 229us) server_negotiation.cc:299] Negotiation successful
0812 02:32:14.772387 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184}
I20260812 02:32:14.773871 14748 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbep4M4/build/debug/bin/kudu with pid 21010
I20260812 02:32:14.782424 14748 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbep4M4/build/debug/bin/kudu with pid 21155
I20260812 02:32:14.793016 14748 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbep4M4/build/debug/bin/kudu with pid 21303
I20260812 02:32:14.800580 14748 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbep4M4/build/debug/bin/kudu with pid 20908
2026-08-12T02:32:14Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2631 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 02:32:14 dist-test-slave-2w3w krb5kdc[21470](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 02:32:14 dist-test-slave-2w3w krb5kdc[21470](info): set up 2 sockets
Aug 12 02:32:14 dist-test-slave-2w3w krb5kdc[21470](info): commencing operation
krb5kdc: starting...
W20260812 02:32:15.997777 14748 mini_kdc.cc:121] Time spent starting KDC: real 1.171s user 0.002s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/krb5kdc/test-user.keytab.
Aug 12 02:32:16 dist-test-slave-2w3w krb5kdc[21470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501936, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T02:32:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T02:32:16Z Disabled control of system clock
WARNING: no policy specified for oryx/127.14.103.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.103.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.103.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/krb5kdc/oryx_127.14.103.62.keytab.
Entry for principal oryx/127.14.103.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/krb5kdc/oryx_127.14.103.62.keytab.
I20260812 02:32:16.162509 14748 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbep4M4/build/debug/bin/kudu
/tmp/dist-test-taskbep4M4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.103.62:37179
--webserver_interface=127.14.103.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/krb5kdc/oryx_127.14.103.62.keytab
--principal=oryx/127.14.103.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.103.20:46127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.103.62:37179
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:32:16.290901 21484 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:32:16.291371 21484 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:32:16.291482 21484 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:32:16.296856 21484 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:32:16.297091 21484 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:32:16.297132 21484 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:32:16.297171 21484 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:32:16.297227 21484 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 02:32:16.308724 21484 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.103.20:46127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.103.62:37179
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/krb5kdc/oryx_127.14.103.62.keytab
--principal=oryx/127.14.103.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.103.62:37179
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.103.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21484
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:32:16.310212 21484 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:32:16.311547 21484 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260812 02:32:16.322774 21484 server_base.cc:1034] running on GCE node
W20260812 02:32:16.322698 21490 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:32:16.322686 21489 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:32:16.322686 21492 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:32:16.324286 21484 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:32:16.326318 21484 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:32:16.327925 21484 hybrid_clock.cc:648] HybridClock initialized: now 1786501936327902 us; error 115 us; skew 500 ppm
Aug 12 02:32:16 dist-test-slave-2w3w krb5kdc[21470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501936, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.103.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:32:16.337270 21484 init.cc:377] Logged in from keytab as oryx/127.14.103.62@KRBTEST.COM (short username oryx)
I20260812 02:32:16.338505 21484 webserver.cc:460] Webserver started at http://127.14.103.62:42363/ using document root <none> and password file <none>
I20260812 02:32:16.339233 21484 fs_manager.cc:359] Metadata directory not provided
I20260812 02:32:16.339342 21484 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:32:16.339607 21484 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:32:16.342566 21484 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/minicluster-data/master-0/data/instance:
uuid: "295ba34a46a74756a017dbe2add34dbb"
format_stamp: "Formatted at 2026-08-12 02:32:16 on dist-test-slave-2w3w"
server_key: "0fe7ea853068518ab08591500c14d50f"
server_key_iv: "43125fc3cd8d5b54912b0fb28f2d06be"
server_key_version: "encryptionkey@0"
I20260812 02:32:16.343315 21484 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/minicluster-data/master-0/wal/instance:
uuid: "295ba34a46a74756a017dbe2add34dbb"
format_stamp: "Formatted at 2026-08-12 02:32:16 on dist-test-slave-2w3w"
server_key: "0fe7ea853068518ab08591500c14d50f"
server_key_iv: "43125fc3cd8d5b54912b0fb28f2d06be"
server_key_version: "encryptionkey@0"
I20260812 02:32:16.349308 21484 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260812 02:32:16.352876 21501 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:32:16.354521 21484 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260812 02:32:16.354734 21484 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/minicluster-data/master-0/data,/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/minicluster-data/master-0/wal
uuid: "295ba34a46a74756a017dbe2add34dbb"
format_stamp: "Formatted at 2026-08-12 02:32:16 on dist-test-slave-2w3w"
server_key: "0fe7ea853068518ab08591500c14d50f"
server_key_iv: "43125fc3cd8d5b54912b0fb28f2d06be"
server_key_version: "encryptionkey@0"
I20260812 02:32:16.354916 21484 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:32:16.381495 21484 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:32:16.385236 21484 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:32:16.385496 21484 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:32:16.394001 21484 rpc_server.cc:307] RPC server started. Bound to: 127.14.103.62:37179
I20260812 02:32:16.394404 21563 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.103.62:37179 every 8 connection(s)
I20260812 02:32:16.395251 21484 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/minicluster-data/master-0/data/info.pb
I20260812 02:32:16.398303 21564 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:32:16.402453 14748 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbep4M4/build/debug/bin/kudu as pid 21484
I20260812 02:32:16.402618 14748 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/minicluster-data/master-0/wal/instance
I20260812 02:32:16.402952 14748 external_mini_cluster.cc:1428] Setting key 25cdc0af1a427ba09aafbb7a263eff25
I20260812 02:32:16.405479 21564 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 295ba34a46a74756a017dbe2add34dbb: Bootstrap starting.
I20260812 02:32:16.408140 21564 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 295ba34a46a74756a017dbe2add34dbb: Neither blocks nor log segments found. Creating new log.
I20260812 02:32:16.408991 21564 log.cc:824] T 00000000000000000000000000000000 P 295ba34a46a74756a017dbe2add34dbb: Log is configured to *not* fsync() on all Append() calls
I20260812 02:32:16.411652 21564 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 295ba34a46a74756a017dbe2add34dbb: No bootstrap required, opened a new log
Aug 12 02:32:16 dist-test-slave-2w3w krb5kdc[21470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501936, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.14.103.62@KRBTEST.COM
I20260812 02:32:16.416766 21564 raft_consensus.cc:348] T 00000000000000000000000000000000 P 295ba34a46a74756a017dbe2add34dbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "295ba34a46a74756a017dbe2add34dbb" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 37179 } }
I20260812 02:32:16.417090 21564 raft_consensus.cc:374] T 00000000000000000000000000000000 P 295ba34a46a74756a017dbe2add34dbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:32:16.417183 21564 raft_consensus.cc:729] T 00000000000000000000000000000000 P 295ba34a46a74756a017dbe2add34dbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 295ba34a46a74756a017dbe2add34dbb, State: Initialized, Role: FOLLOWER
I20260812 02:32:16.417744 21564 consensus_queue.cc:260] T 00000000000000000000000000000000 P 295ba34a46a74756a017dbe2add34dbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "295ba34a46a74756a017dbe2add34dbb" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 37179 } }
I20260812 02:32:16.417918 21564 raft_consensus.cc:388] T 00000000000000000000000000000000 P 295ba34a46a74756a017dbe2add34dbb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 02:32:16.418023 21564 raft_consensus.cc:482] T 00000000000000000000000000000000 P 295ba34a46a74756a017dbe2add34dbb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 02:32:16.418143 21564 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 295ba34a46a74756a017dbe2add34dbb [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:32:16.419477 21564 raft_consensus.cc:504] T 00000000000000000000000000000000 P 295ba34a46a74756a017dbe2add34dbb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "295ba34a46a74756a017dbe2add34dbb" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 37179 } }
I20260812 02:32:16.419873 21564 leader_election.cc:302] T 00000000000000000000000000000000 P 295ba34a46a74756a017dbe2add34dbb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 295ba34a46a74756a017dbe2add34dbb; no voters:
I20260812 02:32:16.420308 21564 leader_election.cc:288] T 00000000000000000000000000000000 P 295ba34a46a74756a017dbe2add34dbb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 02:32:16.420790 21569 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 295ba34a46a74756a017dbe2add34dbb [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:32:16.421037 21569 raft_consensus.cc:686] T 00000000000000000000000000000000 P 295ba34a46a74756a017dbe2add34dbb [term 1 LEADER]: Becoming Leader. State: Replica: 295ba34a46a74756a017dbe2add34dbb, State: Running, Role: LEADER
I20260812 02:32:16.421624 21569 consensus_queue.cc:237] T 00000000000000000000000000000000 P 295ba34a46a74756a017dbe2add34dbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "295ba34a46a74756a017dbe2add34dbb" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 37179 } }
I20260812 02:32:16.422086 21564 sys_catalog.cc:564] T 00000000000000000000000000000000 P 295ba34a46a74756a017dbe2add34dbb [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:32:16.422528 21567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:16.404750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54644 (local address 127.14.103.62:37179)
0812 02:32:16.405435 (+ 685us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:16.405453 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:16.405497 (+ 44us) server_negotiation.cc:407] Connection header received
0812 02:32:16.406564 (+ 1067us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:16.406588 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:16.407054 (+ 466us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:16.407448 (+ 394us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:32:16.408458 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:16.409685 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:16.411098 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:16.411613 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:16.411788 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:16.414992 (+ 3204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:32:16.415023 (+ 31us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:32:16.415050 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:32:16.415097 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:32:16.417860 (+ 2763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:16.418574 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:16.418579 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:16.418585 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:16.418719 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:16.419085 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:16.419088 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:16.419089 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:16.419825 (+ 736us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:32:16.419996 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0812 02:32:16.420687 (+ 691us) server_negotiation.cc:299] Negotiation successful
0812 02:32:16.421094 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":518,"thread_start_us":169,"threads_started":1}
I20260812 02:32:16.425248 21571 sys_catalog.cc:455] T 00000000000000000000000000000000 P 295ba34a46a74756a017dbe2add34dbb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "295ba34a46a74756a017dbe2add34dbb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "295ba34a46a74756a017dbe2add34dbb" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 37179 } } }
I20260812 02:32:16.425295 21570 sys_catalog.cc:455] T 00000000000000000000000000000000 P 295ba34a46a74756a017dbe2add34dbb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 295ba34a46a74756a017dbe2add34dbb. Latest consensus state: current_term: 1 leader_uuid: "295ba34a46a74756a017dbe2add34dbb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "295ba34a46a74756a017dbe2add34dbb" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 37179 } } }
I20260812 02:32:16.425510 21571 sys_catalog.cc:458] T 00000000000000000000000000000000 P 295ba34a46a74756a017dbe2add34dbb [sys.catalog]: This master's current role is: LEADER
I20260812 02:32:16.425513 21570 sys_catalog.cc:458] T 00000000000000000000000000000000 P 295ba34a46a74756a017dbe2add34dbb [sys.catalog]: This master's current role is: LEADER
I20260812 02:32:16.426501 21585 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 02:32:16.430087 21585 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 02:32:16.437146 21585 catalog_manager.cc:1347] Generated new cluster ID: 6e4d29f3e17c4d179f9de01bd0f0be77
I20260812 02:32:16.437256 21585 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 02:32:16.447781 21585 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 02:32:16.448928 21585 catalog_manager.cc:1504] Loading token signing keys...
I20260812 02:32:16.460825 21585 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 295ba34a46a74756a017dbe2add34dbb: Generated new TSK 0
I20260812 02:32:16.461694 21585 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.14.103.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.103.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.103.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/krb5kdc/oryx_127.14.103.1.keytab.
Entry for principal oryx/127.14.103.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/krb5kdc/oryx_127.14.103.1.keytab.
I20260812 02:32:16.512485 14748 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbep4M4/build/debug/bin/kudu
/tmp/dist-test-taskbep4M4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.103.1:0
--local_ip_for_outbound_sockets=127.14.103.1
--webserver_interface=127.14.103.1
--webserver_port=0
--tserver_master_addrs=127.14.103.62:37179
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/krb5kdc/oryx_127.14.103.1.keytab
--principal=oryx/127.14.103.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.103.20:46127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:32:16.634914 21592 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:32:16.635262 21592 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:32:16.635308 21592 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:32:16.639937 21592 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:32:16.640020 21592 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:32:16.640127 21592 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.103.1
I20260812 02:32:16.645869 21592 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.103.20:46127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/krb5kdc/oryx_127.14.103.1.keytab
--principal=oryx/127.14.103.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.103.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.103.1
--webserver_port=0
--tserver_master_addrs=127.14.103.62:37179
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21592
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.103.1
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:32:16.647114 21592 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:32:16.648200 21592 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:32:16.658713 21597 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:32:16.658715 21600 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:32:16.658715 21598 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:32:16.658987 21592 server_base.cc:1034] running on GCE node
I20260812 02:32:16.659655 21592 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:32:16.660439 21592 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:32:16.661660 21592 hybrid_clock.cc:648] HybridClock initialized: now 1786501936661607 us; error 78 us; skew 500 ppm
Aug 12 02:32:16 dist-test-slave-2w3w krb5kdc[21470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501936, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.103.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:32:16.665206 21592 init.cc:377] Logged in from keytab as oryx/127.14.103.1@KRBTEST.COM (short username oryx)
I20260812 02:32:16.666208 21592 webserver.cc:460] Webserver started at http://127.14.103.1:41597/ using document root <none> and password file <none>
I20260812 02:32:16.666805 21592 fs_manager.cc:359] Metadata directory not provided
I20260812 02:32:16.666874 21592 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:32:16.667145 21592 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:32:16.669615 21592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/minicluster-data/ts-0/data/instance:
uuid: "88fa3b5bb4714fa691fcc87e8d327b8f"
format_stamp: "Formatted at 2026-08-12 02:32:16 on dist-test-slave-2w3w"
server_key: "6dbacc41c451a7d910bc00ecae9222d4"
server_key_iv: "bb556621c4acf5ab9cab4bce5da95769"
server_key_version: "encryptionkey@0"
I20260812 02:32:16.670320 21592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/minicluster-data/ts-0/wal/instance:
uuid: "88fa3b5bb4714fa691fcc87e8d327b8f"
format_stamp: "Formatted at 2026-08-12 02:32:16 on dist-test-slave-2w3w"
server_key: "6dbacc41c451a7d910bc00ecae9222d4"
server_key_iv: "bb556621c4acf5ab9cab4bce5da95769"
server_key_version: "encryptionkey@0"
I20260812 02:32:16.675477 21592 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.003s
I20260812 02:32:16.678396 21609 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:32:16.680083 21592 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 02:32:16.680218 21592 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/minicluster-data/ts-0/wal
uuid: "88fa3b5bb4714fa691fcc87e8d327b8f"
format_stamp: "Formatted at 2026-08-12 02:32:16 on dist-test-slave-2w3w"
server_key: "6dbacc41c451a7d910bc00ecae9222d4"
server_key_iv: "bb556621c4acf5ab9cab4bce5da95769"
server_key_version: "encryptionkey@0"
I20260812 02:32:16.680397 21592 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:32:16.709602 21592 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:32:16.713554 21592 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:32:16.713794 21592 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:32:16.714460 21592 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:32:16.715446 21592 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:32:16.715504 21592 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:32:16.715547 21592 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:32:16.715602 21592 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:32:16.725288 21592 rpc_server.cc:307] RPC server started. Bound to: 127.14.103.1:46041
I20260812 02:32:16.725304 21737 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.103.1:46041 every 8 connection(s)
I20260812 02:32:16.726253 21592 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/minicluster-data/ts-0/data/info.pb
I20260812 02:32:16.731693 14748 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbep4M4/build/debug/bin/kudu as pid 21592
I20260812 02:32:16.731849 14748 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/minicluster-data/ts-0/wal/instance
I20260812 02:32:16.732184 14748 external_mini_cluster.cc:1428] Setting key 4790e66bee7b8df33a962ac684b808fe
Aug 12 02:32:16 dist-test-slave-2w3w krb5kdc[21470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501936, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.103.1@KRBTEST.COM for oryx/127.14.103.62@KRBTEST.COM
WARNING: no policy specified for oryx/127.14.103.2@KRBTEST.COM; defaulting to no policy
I20260812 02:32:16.742509 21567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:16.728374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.1:46009 (local address 127.14.103.62:37179)
0812 02:32:16.728520 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:16.728524 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:16.729428 (+ 904us) server_negotiation.cc:407] Connection header received
0812 02:32:16.730821 (+ 1393us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:16.730824 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:16.730878 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:16.731032 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:32:16.732788 (+ 1756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:16.733373 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:16.734627 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:16.734872 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:16.735016 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:16.738054 (+ 3038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:32:16.738061 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:32:16.738067 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:32:16.738106 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:32:16.739999 (+ 1893us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:16.740630 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:16.740634 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:16.740636 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:16.740728 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:16.741124 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:16.741128 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:16.741130 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:16.741340 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:32:16.741454 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0812 02:32:16.742005 (+ 551us) server_negotiation.cc:299] Negotiation successful
0812 02:32:16.742314 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260812 02:32:16.743788 21741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:16.728767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:37179 (local address 127.14.103.1:46009)
0812 02:32:16.729248 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:16.729294 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0812 02:32:16.730576 (+ 1282us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:32:16.731210 (+ 634us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:32:16.731217 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:32:16.731778 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:32:16.732588 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:16.732618 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:16.733529 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:16.733537 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:16.734466 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:16.734479 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:16.735090 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:16.735094 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:16.735363 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:16.736109 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:32:16.736128 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:32:16.737813 (+ 1685us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:32:16.740156 (+ 2343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:32:16.740164 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:32:16.740183 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:32:16.740510 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:32:16.740831 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:32:16.740834 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:32:16.740836 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:32:16.741020 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:32:16.741456 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:32:16.741463 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:32:16.741753 (+ 290us) client_negotiation.cc:769] Sending connection context
0812 02:32:16.741970 (+ 217us) client_negotiation.cc:240] Negotiation successful
0812 02:32:16.742548 (+ 578us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":132,"threads_started":1}
I20260812 02:32:16.745297 21738 heartbeater.cc:347] Connected to a master server at 127.14.103.62:37179
I20260812 02:32:16.745728 21738 heartbeater.cc:464] Registering TS with master...
I20260812 02:32:16.746346 21738 heartbeater.cc:511] Master 127.14.103.62:37179 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.103.2@KRBTEST.COM" created.
I20260812 02:32:16.748267 21520 ts_manager.cc:194] Registered new tserver with Master: 88fa3b5bb4714fa691fcc87e8d327b8f (127.14.103.1:46041)
I20260812 02:32:16.749514 21520 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.14.103.1@KRBTEST.COM'} at 127.14.103.1:46009
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.103.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/krb5kdc/oryx_127.14.103.2.keytab.
Entry for principal oryx/127.14.103.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/krb5kdc/oryx_127.14.103.2.keytab.
I20260812 02:32:16.768033 14748 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbep4M4/build/debug/bin/kudu
/tmp/dist-test-taskbep4M4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.103.2:0
--local_ip_for_outbound_sockets=127.14.103.2
--webserver_interface=127.14.103.2
--webserver_port=0
--tserver_master_addrs=127.14.103.62:37179
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/krb5kdc/oryx_127.14.103.2.keytab
--principal=oryx/127.14.103.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.103.20:46127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:32:16.893532 21744 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:32:16.893898 21744 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:32:16.893976 21744 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:32:16.898691 21744 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:32:16.898852 21744 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:32:16.898954 21744 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.103.2
I20260812 02:32:16.905718 21744 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.103.20:46127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/krb5kdc/oryx_127.14.103.2.keytab
--principal=oryx/127.14.103.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.103.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.103.2
--webserver_port=0
--tserver_master_addrs=127.14.103.62:37179
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21744
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.103.2
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:32:16.907366 21744 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:32:16.908615 21744 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:32:16.920164 21750 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:32:16.920149 21753 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:32:16.920644 21749 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:32:16.922127 21744 server_base.cc:1034] running on GCE node
I20260812 02:32:16.922751 21744 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:32:16.923681 21744 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:32:16.924921 21744 hybrid_clock.cc:648] HybridClock initialized: now 1786501936924820 us; error 115 us; skew 500 ppm
Aug 12 02:32:16 dist-test-slave-2w3w krb5kdc[21470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501936, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.103.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:32:16.929128 21744 init.cc:377] Logged in from keytab as oryx/127.14.103.2@KRBTEST.COM (short username oryx)
I20260812 02:32:16.930395 21744 webserver.cc:460] Webserver started at http://127.14.103.2:35535/ using document root <none> and password file <none>
I20260812 02:32:16.931093 21744 fs_manager.cc:359] Metadata directory not provided
I20260812 02:32:16.931195 21744 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:32:16.931439 21744 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:32:16.933916 21744 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/minicluster-data/ts-1/data/instance:
uuid: "0294bdfc2c3e476fb7c39c969dee950b"
format_stamp: "Formatted at 2026-08-12 02:32:16 on dist-test-slave-2w3w"
server_key: "21a3122387229cbbdf72d316e8dec9b4"
server_key_iv: "57a3b60bc912038b69a3aa6326528f92"
server_key_version: "encryptionkey@0"
I20260812 02:32:16.934603 21744 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/minicluster-data/ts-1/wal/instance:
uuid: "0294bdfc2c3e476fb7c39c969dee950b"
format_stamp: "Formatted at 2026-08-12 02:32:16 on dist-test-slave-2w3w"
server_key: "21a3122387229cbbdf72d316e8dec9b4"
server_key_iv: "57a3b60bc912038b69a3aa6326528f92"
server_key_version: "encryptionkey@0"
I20260812 02:32:16.941696 21744 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260812 02:32:16.945472 21760 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:32:16.947237 21744 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260812 02:32:16.947451 21744 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/minicluster-data/ts-1/wal
uuid: "0294bdfc2c3e476fb7c39c969dee950b"
format_stamp: "Formatted at 2026-08-12 02:32:16 on dist-test-slave-2w3w"
server_key: "21a3122387229cbbdf72d316e8dec9b4"
server_key_iv: "57a3b60bc912038b69a3aa6326528f92"
server_key_version: "encryptionkey@0"
I20260812 02:32:16.947608 21744 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:32:16.967485 21744 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:32:16.971535 21744 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:32:16.971776 21744 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:32:16.972445 21744 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:32:16.973369 21744 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:32:16.973444 21744 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:32:16.973519 21744 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:32:16.973572 21744 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:32:16.985833 21744 rpc_server.cc:307] RPC server started. Bound to: 127.14.103.2:35055
I20260812 02:32:16.985962 21884 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.103.2:35055 every 8 connection(s)
I20260812 02:32:16.986910 21744 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/minicluster-data/ts-1/data/info.pb
I20260812 02:32:16.987280 14748 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbep4M4/build/debug/bin/kudu as pid 21744
I20260812 02:32:16.987466 14748 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/minicluster-data/ts-1/wal/instance
I20260812 02:32:16.987938 14748 external_mini_cluster.cc:1428] Setting key 0b893809ad08b691f558f93cc2f4e39e
Aug 12 02:32:16 dist-test-slave-2w3w krb5kdc[21470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501936, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.103.2@KRBTEST.COM for oryx/127.14.103.62@KRBTEST.COM
WARNING: no policy specified for oryx/127.14.103.3@KRBTEST.COM; defaulting to no policy
I20260812 02:32:17.005088 21567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:16.989608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.2:45839 (local address 127.14.103.62:37179)
0812 02:32:16.989838 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:16.989844 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:16.990743 (+ 899us) server_negotiation.cc:407] Connection header received
0812 02:32:16.992300 (+ 1557us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:16.992306 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:16.992389 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:16.992506 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:32:16.994023 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:16.994577 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:16.996575 (+ 1998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:16.996901 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:16.997042 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:17.000951 (+ 3909us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:32:17.000954 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:32:17.000962 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:32:17.000993 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:32:17.002578 (+ 1585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:17.003317 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:17.003320 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:17.003321 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:17.003404 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:17.003788 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:17.003791 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:17.003793 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:17.004023 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:32:17.004153 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0812 02:32:17.004725 (+ 572us) server_negotiation.cc:299] Negotiation successful
0812 02:32:17.004887 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260812 02:32:17.006016 21887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:16.989686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:37179 (local address 127.14.103.2:45839)
0812 02:32:16.990525 (+ 839us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:16.990593 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0812 02:32:16.992011 (+ 1418us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:32:16.992714 (+ 703us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:32:16.992722 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:32:16.993159 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:32:16.993726 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:16.993742 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:16.994728 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:16.994735 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:16.996370 (+ 1635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:16.996390 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:16.997130 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:16.997138 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:16.997394 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:16.998264 (+ 870us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:32:16.998295 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:32:17.000702 (+ 2407us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:32:17.002736 (+ 2034us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:32:17.002744 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:32:17.002764 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:32:17.003192 (+ 428us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:32:17.003508 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:32:17.003511 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:32:17.003513 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:32:17.003680 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:32:17.004139 (+ 459us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:32:17.004146 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:32:17.004452 (+ 306us) client_negotiation.cc:769] Sending connection context
0812 02:32:17.004664 (+ 212us) client_negotiation.cc:240] Negotiation successful
0812 02:32:17.005055 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":586,"thread_start_us":409,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.103.3@KRBTEST.COM" created.
I20260812 02:32:17.008719 21885 heartbeater.cc:347] Connected to a master server at 127.14.103.62:37179
I20260812 02:32:17.009135 21885 heartbeater.cc:464] Registering TS with master...
I20260812 02:32:17.009929 21885 heartbeater.cc:511] Master 127.14.103.62:37179 requested a full tablet report, sending...
I20260812 02:32:17.011456 21520 ts_manager.cc:194] Registered new tserver with Master: 0294bdfc2c3e476fb7c39c969dee950b (127.14.103.2:35055)
I20260812 02:32:17.012392 21520 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.14.103.2@KRBTEST.COM'} at 127.14.103.2:45839
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.103.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/krb5kdc/oryx_127.14.103.3.keytab.
Entry for principal oryx/127.14.103.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/krb5kdc/oryx_127.14.103.3.keytab.
I20260812 02:32:17.031976 14748 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbep4M4/build/debug/bin/kudu
/tmp/dist-test-taskbep4M4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.103.3:0
--local_ip_for_outbound_sockets=127.14.103.3
--webserver_interface=127.14.103.3
--webserver_port=0
--tserver_master_addrs=127.14.103.62:37179
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/krb5kdc/oryx_127.14.103.3.keytab
--principal=oryx/127.14.103.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.103.20:46127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:32:17.163465 21891 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:32:17.163792 21891 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:32:17.163848 21891 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:32:17.168236 21891 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:32:17.168334 21891 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:32:17.168402 21891 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.103.3
I20260812 02:32:17.174417 21891 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.103.20:46127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/krb5kdc/oryx_127.14.103.3.keytab
--principal=oryx/127.14.103.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.103.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.103.3
--webserver_port=0
--tserver_master_addrs=127.14.103.62:37179
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21891
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.103.3
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:32:17.175743 21891 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:32:17.177172 21891 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:32:17.189929 21898 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:32:17.189929 21897 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:32:17.190049 21902 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:32:17.190356 21891 server_base.cc:1034] running on GCE node
I20260812 02:32:17.190953 21891 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:32:17.192305 21891 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:32:17.193516 21891 hybrid_clock.cc:648] HybridClock initialized: now 1786501937193479 us; error 65 us; skew 500 ppm
Aug 12 02:32:17 dist-test-slave-2w3w krb5kdc[21470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501937, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.103.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:32:17.197023 21891 init.cc:377] Logged in from keytab as oryx/127.14.103.3@KRBTEST.COM (short username oryx)
I20260812 02:32:17.198115 21891 webserver.cc:460] Webserver started at http://127.14.103.3:32941/ using document root <none> and password file <none>
I20260812 02:32:17.198768 21891 fs_manager.cc:359] Metadata directory not provided
I20260812 02:32:17.198876 21891 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:32:17.199174 21891 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:32:17.201583 21891 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/minicluster-data/ts-2/data/instance:
uuid: "9abf9de06df64b46b5db9181d885a31a"
format_stamp: "Formatted at 2026-08-12 02:32:17 on dist-test-slave-2w3w"
server_key: "974bacd2188c10a38b65af0c95095011"
server_key_iv: "f299871e4bf5c14d75f102346ee98d22"
server_key_version: "encryptionkey@0"
I20260812 02:32:17.202278 21891 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/minicluster-data/ts-2/wal/instance:
uuid: "9abf9de06df64b46b5db9181d885a31a"
format_stamp: "Formatted at 2026-08-12 02:32:17 on dist-test-slave-2w3w"
server_key: "974bacd2188c10a38b65af0c95095011"
server_key_iv: "f299871e4bf5c14d75f102346ee98d22"
server_key_version: "encryptionkey@0"
I20260812 02:32:17.207407 21891 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260812 02:32:17.210464 21909 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:32:17.212050 21891 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 02:32:17.212309 21891 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/minicluster-data/ts-2/wal
uuid: "9abf9de06df64b46b5db9181d885a31a"
format_stamp: "Formatted at 2026-08-12 02:32:17 on dist-test-slave-2w3w"
server_key: "974bacd2188c10a38b65af0c95095011"
server_key_iv: "f299871e4bf5c14d75f102346ee98d22"
server_key_version: "encryptionkey@0"
I20260812 02:32:17.212532 21891 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:32:17.224210 21891 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:32:17.228236 21891 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:32:17.228597 21891 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:32:17.229364 21891 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:32:17.230331 21891 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:32:17.230412 21891 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:32:17.230487 21891 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:32:17.230530 21891 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:32:17.241041 21891 rpc_server.cc:307] RPC server started. Bound to: 127.14.103.3:36861
I20260812 02:32:17.241058 22037 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.103.3:36861 every 8 connection(s)
I20260812 02:32:17.242460 21891 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/minicluster-data/ts-2/data/info.pb
I20260812 02:32:17.251994 14748 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbep4M4/build/debug/bin/kudu as pid 21891
I20260812 02:32:17.252140 14748 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/minicluster-data/ts-2/wal/instance
I20260812 02:32:17.252516 14748 external_mini_cluster.cc:1428] Setting key bd6186f832a63a89a14f8526bf237a3b
Aug 12 02:32:17 dist-test-slave-2w3w krb5kdc[21470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501937, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.103.3@KRBTEST.COM for oryx/127.14.103.62@KRBTEST.COM
I20260812 02:32:17.258935 21567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:17.245021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.3:41135 (local address 127.14.103.62:37179)
0812 02:32:17.245272 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:17.245276 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:17.246091 (+ 815us) server_negotiation.cc:407] Connection header received
0812 02:32:17.247654 (+ 1563us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:17.247658 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:17.247722 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:17.247833 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:32:17.249085 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:17.249627 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:17.250844 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:17.251129 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:17.251246 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:17.254766 (+ 3520us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:32:17.254769 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:32:17.254771 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:32:17.254801 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:32:17.256571 (+ 1770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:17.257189 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:17.257194 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:17.257196 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:17.257290 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:17.257680 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:17.257684 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:17.257686 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:17.257903 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:32:17.258040 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0812 02:32:17.258614 (+ 574us) server_negotiation.cc:299] Negotiation successful
0812 02:32:17.258760 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136}
I20260812 02:32:17.259907 22040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:17.245379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:37179 (local address 127.14.103.3:41135)
0812 02:32:17.245917 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:17.245965 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0812 02:32:17.247428 (+ 1463us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:32:17.248023 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:32:17.248030 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:32:17.248432 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:32:17.248931 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:17.248944 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:17.249763 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:17.249773 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:17.250687 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:17.250697 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:17.251306 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:17.251311 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:17.251493 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:17.252418 (+ 925us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:32:17.252443 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:32:17.254534 (+ 2091us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:32:17.256719 (+ 2185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:32:17.256727 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:32:17.256747 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:32:17.257070 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:32:17.257393 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:32:17.257397 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:32:17.257399 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:32:17.257572 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:32:17.258040 (+ 468us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:32:17.258049 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:32:17.258345 (+ 296us) client_negotiation.cc:769] Sending connection context
0812 02:32:17.258569 (+ 224us) client_negotiation.cc:240] Negotiation successful
0812 02:32:17.258939 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":141,"threads_started":1}
I20260812 02:32:17.261300 22038 heartbeater.cc:347] Connected to a master server at 127.14.103.62:37179
I20260812 02:32:17.261567 22038 heartbeater.cc:464] Registering TS with master...
I20260812 02:32:17.262063 22038 heartbeater.cc:511] Master 127.14.103.62:37179 requested a full tablet report, sending...
I20260812 02:32:17.263324 21520 ts_manager.cc:194] Registered new tserver with Master: 9abf9de06df64b46b5db9181d885a31a (127.14.103.3:36861)
I20260812 02:32:17.263906 21520 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.14.103.3@KRBTEST.COM'} at 127.14.103.3:41135
I20260812 02:32:17.267596 14748 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.14.103.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.103.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.103.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/krb5kdc/oryx_127.14.103.61.keytab.
Entry for principal oryx/127.14.103.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/krb5kdc/oryx_127.14.103.61.keytab.
I20260812 02:32:17.303562 14748 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbep4M4/build/debug/bin/kudu
/tmp/dist-test-taskbep4M4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.103.61:35785
--webserver_interface=127.14.103.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/krb5kdc/oryx_127.14.103.61.keytab
--principal=oryx/127.14.103.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.103.20:46127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.103.62:37179,127.14.103.61:35785
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:32:17.427983 22043 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:32:17.428328 22043 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:32:17.428372 22043 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:32:17.432839 22043 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:32:17.432927 22043 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:32:17.432948 22043 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:32:17.432969 22043 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:32:17.432986 22043 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 02:32:17.439752 22043 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.103.20:46127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.14.103.62:37179,127.14.103.61:35785
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/krb5kdc/oryx_127.14.103.61.keytab
--principal=oryx/127.14.103.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.103.61:35785
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.14.103.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22043
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
W20260812 02:32:17.440047 22043 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.14.103.62:37179,127.14.103.61:35785'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260812 02:32:17.441008 22043 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:32:17.442162 22043 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:32:17.451587 22048 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:32:17.451659 22049 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:32:17.451599 22051 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:32:17.451719 22043 server_base.cc:1034] running on GCE node
I20260812 02:32:17.452559 22043 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:32:17.453835 22043 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:32:17.455085 22043 hybrid_clock.cc:648] HybridClock initialized: now 1786501937455068 us; error 44 us; skew 500 ppm
Aug 12 02:32:17 dist-test-slave-2w3w krb5kdc[21470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501937, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.103.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:32:17.458288 22043 init.cc:377] Logged in from keytab as oryx/127.14.103.61@KRBTEST.COM (short username oryx)
I20260812 02:32:17.459431 22043 webserver.cc:460] Webserver started at http://127.14.103.61:42377/ using document root <none> and password file <none>
I20260812 02:32:17.460101 22043 fs_manager.cc:359] Metadata directory not provided
I20260812 02:32:17.460172 22043 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:32:17.460361 22043 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:32:17.462842 22043 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/minicluster-data/master-1/data/instance:
uuid: "198952c9f5644bc5a514d3f6d8a6a871"
format_stamp: "Formatted at 2026-08-12 02:32:17 on dist-test-slave-2w3w"
server_key: "19bcdd20c6149dc79255cca5ffd1e3f5"
server_key_iv: "adabe4a935d0c144015dd82f1528443a"
server_key_version: "encryptionkey@0"
I20260812 02:32:17.463529 22043 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/minicluster-data/master-1/wal/instance:
uuid: "198952c9f5644bc5a514d3f6d8a6a871"
format_stamp: "Formatted at 2026-08-12 02:32:17 on dist-test-slave-2w3w"
server_key: "19bcdd20c6149dc79255cca5ffd1e3f5"
server_key_iv: "adabe4a935d0c144015dd82f1528443a"
server_key_version: "encryptionkey@0"
I20260812 02:32:17.467969 22043 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260812 02:32:17.471199 22059 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:32:17.472729 22043 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 02:32:17.472877 22043 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/minicluster-data/master-1/data,/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/minicluster-data/master-1/wal
uuid: "198952c9f5644bc5a514d3f6d8a6a871"
format_stamp: "Formatted at 2026-08-12 02:32:17 on dist-test-slave-2w3w"
server_key: "19bcdd20c6149dc79255cca5ffd1e3f5"
server_key_iv: "adabe4a935d0c144015dd82f1528443a"
server_key_version: "encryptionkey@0"
I20260812 02:32:17.472995 22043 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:32:17.503759 22043 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:32:17.508030 22043 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:32:17.508225 22043 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:32:17.516413 22043 rpc_server.cc:307] RPC server started. Bound to: 127.14.103.61:35785
I20260812 02:32:17.516758 22121 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.103.61:35785 every 8 connection(s)
I20260812 02:32:17.517671 22043 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/minicluster-data/master-1/data/info.pb
I20260812 02:32:17.521580 22122 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:32:17.523066 14748 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbep4M4/build/debug/bin/kudu as pid 22043
I20260812 02:32:17.526777 22122 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.103.62" port: 37179 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 02:32:17 dist-test-slave-2w3w krb5kdc[21470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501936, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.14.103.61@KRBTEST.COM
Aug 12 02:32:17 dist-test-slave-2w3w krb5kdc[21470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501937, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.103.61@KRBTEST.COM for oryx/127.14.103.62@KRBTEST.COM
I20260812 02:32:17.540092 21567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:17.528881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54652 (local address 127.14.103.62:37179)
0812 02:32:17.529032 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:17.529037 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:17.529492 (+ 455us) server_negotiation.cc:407] Connection header received
0812 02:32:17.529702 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:17.529706 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:17.529763 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:17.529852 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:32:17.530634 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:17.531287 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:17.532215 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:17.532682 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:17.532808 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:17.535744 (+ 2936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:32:17.535749 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:32:17.535752 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:32:17.535798 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:32:17.537614 (+ 1816us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:17.538300 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:17.538304 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:17.538305 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:17.538402 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:17.538878 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:17.538884 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:17.538888 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:17.539224 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:32:17.539329 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0812 02:32:17.539780 (+ 451us) server_negotiation.cc:299] Negotiation successful
0812 02:32:17.539918 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260812 02:32:17.540753 22125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:17.524888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34234 (local address 127.14.103.61:35785)
0812 02:32:17.525576 (+ 688us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:17.525593 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:17.525633 (+ 40us) server_negotiation.cc:407] Connection header received
0812 02:32:17.526304 (+ 671us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:17.526322 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:17.526718 (+ 396us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:17.527078 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:32:17.528042 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:17.529337 (+ 1295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:17.530295 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:17.530923 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:17.531144 (+ 221us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:17.534334 (+ 3190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:32:17.534342 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:32:17.534356 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:32:17.534389 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:32:17.536876 (+ 2487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:17.537479 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:17.537486 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:17.537496 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:17.537652 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:17.538059 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:17.538062 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:17.538063 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:17.538704 (+ 641us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:32:17.538939 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0812 02:32:17.539396 (+ 457us) server_negotiation.cc:299] Negotiation successful
0812 02:32:17.539751 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":469,"thread_start_us":309,"threads_started":1}
I20260812 02:32:17.540753 22126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:17.528809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:37179 (local address 127.0.0.1:54652)
0812 02:32:17.529356 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:17.529412 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0812 02:32:17.529586 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:32:17.529903 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:32:17.529908 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:32:17.530349 (+ 441us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:32:17.530503 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:17.530510 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:17.531422 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:17.531425 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:17.532081 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:17.532090 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:17.532877 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:17.532881 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:17.532994 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:17.533603 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:32:17.533620 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:32:17.535494 (+ 1874us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:32:17.537795 (+ 2301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:32:17.537805 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:32:17.537812 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:32:17.538175 (+ 363us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:32:17.538630 (+ 455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:32:17.538633 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:32:17.538634 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:32:17.538726 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:32:17.539379 (+ 653us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:32:17.539387 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:32:17.539530 (+ 143us) client_negotiation.cc:769] Sending connection context
0812 02:32:17.539702 (+ 172us) client_negotiation.cc:240] Negotiation successful
0812 02:32:17.539909 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":263,"threads_started":1}
I20260812 02:32:17.542618 22122 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.103.61" port: 35785 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 02:32:17 dist-test-slave-2w3w krb5kdc[21470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501937, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.103.61@KRBTEST.COM for oryx/127.14.103.61@KRBTEST.COM
I20260812 02:32:17.552243 22126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:17.543220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.61:35785 (local address 127.0.0.1:34236)
0812 02:32:17.543390 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:17.543407 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 02:32:17.543497 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:32:17.543743 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:32:17.543746 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:32:17.543942 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:32:17.544031 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:17.544036 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:17.545127 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:17.545133 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:17.545649 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:17.545656 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:17.546110 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:17.546113 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:17.546203 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:17.546702 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:32:17.546714 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:32:17.548147 (+ 1433us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:32:17.550411 (+ 2264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:32:17.550415 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:32:17.550417 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:32:17.550633 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:32:17.551100 (+ 467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:32:17.551105 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:32:17.551109 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:32:17.551226 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:32:17.551762 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:32:17.551765 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:32:17.551857 (+ 92us) client_negotiation.cc:769] Sending connection context
0812 02:32:17.551973 (+ 116us) client_negotiation.cc:240] Negotiation successful
0812 02:32:17.552102 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86}
I20260812 02:32:17.552567 22125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:17.543355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34236 (local address 127.14.103.61:35785)
0812 02:32:17.543471 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:17.543476 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:17.543490 (+ 14us) server_negotiation.cc:407] Connection header received
0812 02:32:17.543582 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:17.543584 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:17.543631 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:17.543733 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:32:17.544137 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:17.545008 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:17.545765 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:17.545999 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:17.546108 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:17.548305 (+ 2197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:32:17.548309 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:32:17.548310 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:32:17.548330 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:32:17.550158 (+ 1828us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:17.550756 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:17.550760 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:17.550762 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:17.550845 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:17.551352 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:17.551356 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:17.551357 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:17.551659 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:32:17.551747 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0812 02:32:17.552122 (+ 375us) server_negotiation.cc:299] Negotiation successful
0812 02:32:17.552412 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260812 02:32:17.556210 22122 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 198952c9f5644bc5a514d3f6d8a6a871: Bootstrap starting.
I20260812 02:32:17.559333 22122 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 198952c9f5644bc5a514d3f6d8a6a871: Neither blocks nor log segments found. Creating new log.
I20260812 02:32:17.560214 22122 log.cc:824] T 00000000000000000000000000000000 P 198952c9f5644bc5a514d3f6d8a6a871: Log is configured to *not* fsync() on all Append() calls
I20260812 02:32:17.562850 22122 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 198952c9f5644bc5a514d3f6d8a6a871: No bootstrap required, opened a new log
I20260812 02:32:17.566354 22122 raft_consensus.cc:348] T 00000000000000000000000000000000 P 198952c9f5644bc5a514d3f6d8a6a871 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "295ba34a46a74756a017dbe2add34dbb" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 37179 } } peers { permanent_uuid: "198952c9f5644bc5a514d3f6d8a6a871" member_type: VOTER last_known_addr { host: "127.14.103.61" port: 35785 } }
I20260812 02:32:17.566685 22122 raft_consensus.cc:374] T 00000000000000000000000000000000 P 198952c9f5644bc5a514d3f6d8a6a871 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:32:17.566735 22122 raft_consensus.cc:729] T 00000000000000000000000000000000 P 198952c9f5644bc5a514d3f6d8a6a871 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 198952c9f5644bc5a514d3f6d8a6a871, State: Initialized, Role: FOLLOWER
I20260812 02:32:17.567301 22122 consensus_queue.cc:260] T 00000000000000000000000000000000 P 198952c9f5644bc5a514d3f6d8a6a871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "295ba34a46a74756a017dbe2add34dbb" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 37179 } } peers { permanent_uuid: "198952c9f5644bc5a514d3f6d8a6a871" member_type: VOTER last_known_addr { host: "127.14.103.61" port: 35785 } }
I20260812 02:32:17.568401 22128 sys_catalog.cc:455] T 00000000000000000000000000000000 P 198952c9f5644bc5a514d3f6d8a6a871 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "295ba34a46a74756a017dbe2add34dbb" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 37179 } } peers { permanent_uuid: "198952c9f5644bc5a514d3f6d8a6a871" member_type: VOTER last_known_addr { host: "127.14.103.61" port: 35785 } } }
I20260812 02:32:17.568584 22128 sys_catalog.cc:458] T 00000000000000000000000000000000 P 198952c9f5644bc5a514d3f6d8a6a871 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 02:32:17.568630 22122 sys_catalog.cc:564] T 00000000000000000000000000000000 P 198952c9f5644bc5a514d3f6d8a6a871 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:32:17.570705 22128 raft_consensus.cc:482] T 00000000000000000000000000000000 P 198952c9f5644bc5a514d3f6d8a6a871 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 02:32:17.571491 22128 raft_consensus.cc:504] T 00000000000000000000000000000000 P 198952c9f5644bc5a514d3f6d8a6a871 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "295ba34a46a74756a017dbe2add34dbb" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 37179 } } peers { permanent_uuid: "198952c9f5644bc5a514d3f6d8a6a871" member_type: VOTER last_known_addr { host: "127.14.103.61" port: 35785 } }
I20260812 02:32:17.572659 22128 leader_election.cc:288] T 00000000000000000000000000000000 P 198952c9f5644bc5a514d3f6d8a6a871 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 295ba34a46a74756a017dbe2add34dbb (127.14.103.62:37179)
I20260812 02:32:17.573215 21531 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "198952c9f5644bc5a514d3f6d8a6a871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "295ba34a46a74756a017dbe2add34dbb" is_pre_election: true
I20260812 02:32:17.574267 22061 leader_election.cc:302] T 00000000000000000000000000000000 P 198952c9f5644bc5a514d3f6d8a6a871 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 198952c9f5644bc5a514d3f6d8a6a871; no voters: 295ba34a46a74756a017dbe2add34dbb
I20260812 02:32:17.574585 22128 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 198952c9f5644bc5a514d3f6d8a6a871 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260812 02:32:17.577003 22139 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 198952c9f5644bc5a514d3f6d8a6a871: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 02:32:17.577131 22139 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 02:32:17.577953 22043 master_runner.cc:428] Detected that this master 198952c9f5644bc5a514d3f6d8a6a871 is joining an existing cluster
I20260812 02:32:17.578033 22043 master_runner.cc:432] Initiating AddMaster RPC to add 127.14.103.61:35785
I20260812 02:32:17.592703 21567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:17.582674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54666 (local address 127.14.103.62:37179)
0812 02:32:17.583230 (+ 556us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:17.583236 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:17.583256 (+ 20us) server_negotiation.cc:407] Connection header received
0812 02:32:17.583310 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:17.583313 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:17.583369 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:17.583478 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:32:17.584375 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:17.585232 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:17.586570 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:17.586928 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:17.587061 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:17.588642 (+ 1581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:32:17.588648 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:32:17.588654 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:32:17.588692 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:32:17.590434 (+ 1742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:17.591072 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:17.591076 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:17.591078 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:17.591184 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:17.591582 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:17.591586 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:17.591588 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:17.591859 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:32:17.591967 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0812 02:32:17.592370 (+ 403us) server_negotiation.cc:299] Negotiation successful
0812 02:32:17.592519 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":454}
I20260812 02:32:17.592712 22149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:17.581463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:37179 (local address 127.0.0.1:54666)
0812 02:32:17.582287 (+ 824us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:17.582303 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 02:32:17.582419 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:32:17.583640 (+ 1221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:32:17.583644 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:32:17.583921 (+ 277us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:32:17.584098 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:17.584109 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:17.585446 (+ 1337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:17.585450 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:17.586401 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:17.586414 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:17.587246 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:17.587250 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:17.587346 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:17.587929 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:32:17.587944 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:32:17.588471 (+ 527us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:32:17.590585 (+ 2114us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:32:17.590589 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:32:17.590591 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:32:17.590868 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:32:17.591299 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:32:17.591303 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:32:17.591305 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:32:17.591470 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:32:17.591999 (+ 529us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:32:17.592002 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:32:17.592112 (+ 110us) client_negotiation.cc:769] Sending connection context
0812 02:32:17.592282 (+ 170us) client_negotiation.cc:240] Negotiation successful
0812 02:32:17.592471 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":719,"spinlock_wait_cycles":1408,"thread_start_us":178,"threads_started":1}
I20260812 02:32:17.593132 22125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:17.581637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34238 (local address 127.14.103.61:35785)
0812 02:32:17.581763 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:17.581768 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:17.583521 (+ 1753us) server_negotiation.cc:407] Connection header received
0812 02:32:17.584195 (+ 674us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:17.584199 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:17.584265 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:17.584391 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:32:17.585169 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:17.585771 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:17.586686 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:17.586915 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:17.587072 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:17.588400 (+ 1328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:32:17.588404 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:32:17.588407 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:32:17.588430 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:32:17.590434 (+ 2004us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:17.591385 (+ 951us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:17.591389 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:17.591390 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:17.591470 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:17.592225 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:17.592230 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:17.592234 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:17.592470 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:32:17.592565 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0812 02:32:17.592832 (+ 267us) server_negotiation.cc:299] Negotiation successful
0812 02:32:17.592956 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49,"spinlock_wait_cycles":4608}
I20260812 02:32:17.593569 22147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:17.581468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.61:35785 (local address 127.0.0.1:34238)
0812 02:32:17.583124 (+ 1656us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:17.583345 (+ 221us) client_negotiation.cc:174] Beginning negotiation
0812 02:32:17.583936 (+ 591us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:32:17.584409 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:32:17.584412 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:32:17.584846 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:32:17.585000 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:17.585008 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:17.585908 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:17.585913 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:17.586561 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:17.586571 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:17.587092 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:17.587095 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:17.587191 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:17.587759 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:32:17.587780 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:32:17.588252 (+ 472us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:32:17.591037 (+ 2785us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:32:17.591040 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:32:17.591042 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:32:17.591277 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:32:17.592018 (+ 741us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:32:17.592021 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:32:17.592023 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:32:17.592117 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:32:17.592598 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:32:17.592601 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:32:17.592671 (+ 70us) client_negotiation.cc:769] Sending connection context
0812 02:32:17.593280 (+ 609us) client_negotiation.cc:240] Negotiation successful
0812 02:32:17.593448 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1194,"mutex_wait_us":315,"thread_start_us":320,"threads_started":1}
WARNING: no policy specified for oryx/127.14.103.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.103.60@KRBTEST.COM" created.
Aug 12 02:32:17 dist-test-slave-2w3w krb5kdc[21470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501936, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.103.62@KRBTEST.COM for oryx/127.14.103.61@KRBTEST.COM
I20260812 02:32:17.615682 22152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:17.599693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.61:35785 (local address 127.0.0.1:34246)
0812 02:32:17.603537 (+ 3844us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:17.603630 (+ 93us) client_negotiation.cc:174] Beginning negotiation
0812 02:32:17.603868 (+ 238us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:32:17.604320 (+ 452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:32:17.604330 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:32:17.604882 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:32:17.605067 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:17.605078 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:17.606169 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:17.606176 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:17.606888 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:17.606900 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:17.607581 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:17.607585 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:17.607674 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:17.608465 (+ 791us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:32:17.608486 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:32:17.611220 (+ 2734us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:32:17.613275 (+ 2055us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:32:17.613294 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:32:17.613302 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:32:17.613733 (+ 431us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:32:17.614112 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:32:17.614115 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:32:17.614118 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:32:17.614221 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:32:17.614864 (+ 643us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:32:17.614873 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:32:17.615061 (+ 188us) client_negotiation.cc:769] Sending connection context
0812 02:32:17.615219 (+ 158us) client_negotiation.cc:240] Negotiation successful
0812 02:32:17.615479 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3538,"thread_start_us":101,"threads_started":1}
I20260812 02:32:17.615686 22125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:17.599701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34246 (local address 127.14.103.61:35785)
0812 02:32:17.599832 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:17.599836 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:17.603772 (+ 3936us) server_negotiation.cc:407] Connection header received
0812 02:32:17.604061 (+ 289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:17.604064 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:17.604134 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:17.604236 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:32:17.605223 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:17.606027 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:17.607097 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:17.607440 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:17.607559 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:17.611428 (+ 3869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:32:17.611432 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:32:17.611435 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:32:17.611466 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:32:17.613101 (+ 1635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:17.613878 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:17.613883 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:17.613886 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:17.614000 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:17.614404 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:17.614410 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:17.614413 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:17.614722 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:32:17.614853 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0812 02:32:17.615328 (+ 475us) server_negotiation.cc:299] Negotiation successful
0812 02:32:17.615481 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260812 02:32:17.617890 21520 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.14.103.61:35785: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "198952c9f5644bc5a514d3f6d8a6a871"
member_type: NON_VOTER
last_known_addr {
host: "127.14.103.61"
port: 35785
}
attrs {
promote: true
}
}
dest_uuid: "295ba34a46a74756a017dbe2add34dbb"
cas_config_opid_index: -1
I20260812 02:32:17.619272 21520 consensus_queue.cc:237] T 00000000000000000000000000000000 P 295ba34a46a74756a017dbe2add34dbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "295ba34a46a74756a017dbe2add34dbb" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 37179 } } peers { permanent_uuid: "198952c9f5644bc5a514d3f6d8a6a871" member_type: NON_VOTER last_known_addr { host: "127.14.103.61" port: 35785 } attrs { promote: true } }
I20260812 02:32:17.621618 22156 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 295ba34a46a74756a017dbe2add34dbb [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 198952c9f5644bc5a514d3f6d8a6a871 (127.14.103.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "295ba34a46a74756a017dbe2add34dbb" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 37179 } } peers { permanent_uuid: "198952c9f5644bc5a514d3f6d8a6a871" member_type: NON_VOTER last_known_addr { host: "127.14.103.61" port: 35785 } attrs { promote: true } } }
I20260812 02:32:17.622537 22091 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 198952c9f5644bc5a514d3f6d8a6a871 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:32:17.625047 22156 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.14.103.61:35785
I20260812 02:32:17.625206 22091 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 198952c9f5644bc5a514d3f6d8a6a871 [term 1 FOLLOWER]: Refusing update from remote peer 295ba34a46a74756a017dbe2add34dbb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260812 02:32:17.625662 22155 sys_catalog.cc:455] T 00000000000000000000000000000000 P 295ba34a46a74756a017dbe2add34dbb [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "295ba34a46a74756a017dbe2add34dbb" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "295ba34a46a74756a017dbe2add34dbb" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 37179 } } peers { permanent_uuid: "198952c9f5644bc5a514d3f6d8a6a871" member_type: NON_VOTER last_known_addr { host: "127.14.103.61" port: 35785 } attrs { promote: true } } }
I20260812 02:32:17.625829 22155 sys_catalog.cc:458] T 00000000000000000000000000000000 P 295ba34a46a74756a017dbe2add34dbb [sys.catalog]: This master's current role is: LEADER
I20260812 02:32:17.626302 22043 master.cc:537] Master@127.14.103.61:35785 shutting down...
I20260812 02:32:17.627308 22155 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 295ba34a46a74756a017dbe2add34dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "198952c9f5644bc5a514d3f6d8a6a871" member_type: NON_VOTER last_known_addr { host: "127.14.103.61" port: 35785 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260812 02:32:17.632011 21503 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260812 02:32:17.632742 22043 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 198952c9f5644bc5a514d3f6d8a6a871 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 02:32:17.633111 22043 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 198952c9f5644bc5a514d3f6d8a6a871 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:32:17.633240 22043 tablet_replica.cc:330] T 00000000000000000000000000000000 P 198952c9f5644bc5a514d3f6d8a6a871: stopping tablet replica
W20260812 02:32:17.633535 21503 consensus_peers.cc:487] T 00000000000000000000000000000000 P 295ba34a46a74756a017dbe2add34dbb -> Peer 198952c9f5644bc5a514d3f6d8a6a871 (127.14.103.61:35785): Couldn't send request to peer 198952c9f5644bc5a514d3f6d8a6a871. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.103.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/krb5kdc/oryx_127.14.103.60.keytab.
Entry for principal oryx/127.14.103.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/krb5kdc/oryx_127.14.103.60.keytab.
I20260812 02:32:17.638935 14748 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbep4M4/build/debug/bin/kudu
/tmp/dist-test-taskbep4M4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.103.60:40443
--webserver_interface=127.14.103.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/krb5kdc/oryx_127.14.103.60.keytab
--principal=oryx/127.14.103.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.103.20:46127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.103.62:37179,127.14.103.61:35785,127.14.103.60:40443
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 02:32:17.640123 22043 master.cc:552] Master@127.14.103.61:35785 shutdown complete.
I20260812 02:32:17.640233 22043 master_runner.cc:305] Clearing existing system tablet
I20260812 02:32:17.640538 22043 fs_manager.cc:359] Metadata directory not provided
I20260812 02:32:17.640643 22043 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:32:17.642150 22043 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260812 02:32:17.643217 22163 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:32:17.643603 22043 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:32:17.643740 22043 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 02:32:17.643776 22043 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/minicluster-data/master-1/data,/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/minicluster-data/master-1/wal
uuid: "198952c9f5644bc5a514d3f6d8a6a871"
format_stamp: "Formatted at 2026-08-12 02:32:17 on dist-test-slave-2w3w"
server_key: "19bcdd20c6149dc79255cca5ffd1e3f5"
server_key_iv: "adabe4a935d0c144015dd82f1528443a"
server_key_version: "encryptionkey@0"
I20260812 02:32:17.646037 22043 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 198952c9f5644bc5a514d3f6d8a6a871: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:32:17.648748 22043 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 198952c9f5644bc5a514d3f6d8a6a871: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260812 02:32:17.648854 22043 log.cc:1196] T 00000000000000000000000000000000 P 198952c9f5644bc5a514d3f6d8a6a871: Deleting WAL directory at /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260812 02:32:17.649127 22043 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 198952c9f5644bc5a514d3f6d8a6a871: Deleting consensus metadata
I20260812 02:32:17.649447 22043 master_runner.cc:315] Copying system tablet from 127.14.103.62:37179
I20260812 02:32:17.650920 22043 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 198952c9f5644bc5a514d3f6d8a6a871: tablet copy: Beginning tablet copy session from remote peer at address 127.14.103.62:37179
I20260812 02:32:17.661778 21567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:17.651955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54680 (local address 127.14.103.62:37179)
0812 02:32:17.652090 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:17.652095 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:17.652358 (+ 263us) server_negotiation.cc:407] Connection header received
0812 02:32:17.652589 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:17.652596 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:17.652673 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:17.652782 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:32:17.653560 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:17.654422 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:17.655721 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:17.656087 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:17.656203 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:17.657861 (+ 1658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:32:17.657866 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:32:17.657869 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:32:17.657899 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:32:17.659763 (+ 1864us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:17.660384 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:17.660388 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:17.660390 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:17.660495 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:17.660840 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:17.660843 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:17.660844 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:17.661079 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:32:17.661179 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0812 02:32:17.661483 (+ 304us) server_negotiation.cc:299] Negotiation successful
0812 02:32:17.661612 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260812 02:32:17.661798 22171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:17.651811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:37179 (local address 127.0.0.1:54680)
0812 02:32:17.652230 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:17.652245 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 02:32:17.652447 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:32:17.652816 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:32:17.652819 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:32:17.653083 (+ 264us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:32:17.653212 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:17.653219 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:17.654572 (+ 1353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:17.654576 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:17.655583 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:17.655595 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:17.656468 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:17.656472 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:17.656565 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:17.657206 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:32:17.657222 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:32:17.657689 (+ 467us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:32:17.659913 (+ 2224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:32:17.659916 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:32:17.659918 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:32:17.660266 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:32:17.660630 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:32:17.660633 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:32:17.660635 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:32:17.660733 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:32:17.661185 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:32:17.661188 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:32:17.661295 (+ 107us) client_negotiation.cc:769] Sending connection context
0812 02:32:17.661408 (+ 113us) client_negotiation.cc:240] Negotiation successful
0812 02:32:17.661566 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":105,"threads_started":1}
I20260812 02:32:17.662911 21545 tablet_copy_service.cc:140] P 295ba34a46a74756a017dbe2add34dbb: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 198952c9f5644bc5a514d3f6d8a6a871 ({username='oryx', principal='oryx/127.14.103.61@KRBTEST.COM'} at 127.0.0.1:54680)
I20260812 02:32:17.663110 21545 tablet_copy_service.cc:161] P 295ba34a46a74756a017dbe2add34dbb: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 198952c9f5644bc5a514d3f6d8a6a871 at {username='oryx', principal='oryx/127.14.103.61@KRBTEST.COM'} at 127.0.0.1:54680: session id = 198952c9f5644bc5a514d3f6d8a6a871-00000000000000000000000000000000
I20260812 02:32:17.665163 21545 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 295ba34a46a74756a017dbe2add34dbb: Tablet Copy: opened 0 blocks and 1 log segments
I20260812 02:32:17.667352 22043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260812 02:32:17.670449 22043 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 198952c9f5644bc5a514d3f6d8a6a871: tablet copy: Starting download of 0 data blocks...
I20260812 02:32:17.670745 22043 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 198952c9f5644bc5a514d3f6d8a6a871: tablet copy: Starting download of 1 WAL segments...
I20260812 02:32:17.673131 22043 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 198952c9f5644bc5a514d3f6d8a6a871: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260812 02:32:17.675638 21545 tablet_copy_service.cc:342] P 295ba34a46a74756a017dbe2add34dbb: Request end of tablet copy session 198952c9f5644bc5a514d3f6d8a6a871-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.14.103.61@KRBTEST.COM'} at 127.0.0.1:54680
I20260812 02:32:17.675783 21545 tablet_copy_service.cc:433] P 295ba34a46a74756a017dbe2add34dbb: ending tablet copy session 198952c9f5644bc5a514d3f6d8a6a871-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 198952c9f5644bc5a514d3f6d8a6a871
I20260812 02:32:17.685405 22043 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:32:17.687582 22172 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:32:17.687980 22175 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:32:17.688410 22173 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:32:17.688477 22043 server_base.cc:1034] running on GCE node
I20260812 02:32:17.689090 22043 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:32:17.689558 22043 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:32:17.690732 22043 hybrid_clock.cc:648] HybridClock initialized: now 1786501937690719 us; error 38 us; skew 500 ppm
Aug 12 02:32:17 dist-test-slave-2w3w krb5kdc[21470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501937, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.103.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:32:17.692507 22043 init.cc:377] Logged in from keytab as oryx/127.14.103.61@KRBTEST.COM (short username oryx)
I20260812 02:32:17.693097 22043 webserver.cc:460] Webserver started at http://127.14.103.61:37565/ using document root <none> and password file <none>
I20260812 02:32:17.693315 22043 fs_manager.cc:359] Metadata directory not provided
I20260812 02:32:17.693399 22043 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:32:17.695060 22043 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260812 02:32:17.696246 22182 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:32:17.696586 22043 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260812 02:32:17.696667 22043 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/minicluster-data/master-1/data,/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/minicluster-data/master-1/wal
uuid: "198952c9f5644bc5a514d3f6d8a6a871"
format_stamp: "Formatted at 2026-08-12 02:32:17 on dist-test-slave-2w3w"
server_key: "19bcdd20c6149dc79255cca5ffd1e3f5"
server_key_iv: "adabe4a935d0c144015dd82f1528443a"
server_key_version: "encryptionkey@0"
I20260812 02:32:17.696756 22043 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:32:17.713338 22043 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:32:17.715988 22043 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:32:17.721527 22043 rpc_server.cc:307] RPC server started. Bound to: 127.14.103.61:35785
I20260812 02:32:17.721575 22241 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.103.61:35785 every 8 connection(s)
I20260812 02:32:17.722304 22043 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/minicluster-data/master-1/data/info.pb
I20260812 02:32:17.725179 22242 sys_catalog.cc:263] Verifying existing consensus state
I20260812 02:32:17.726420 22242 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 198952c9f5644bc5a514d3f6d8a6a871: Bootstrap starting.
I20260812 02:32:17.738410 22242 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 198952c9f5644bc5a514d3f6d8a6a871: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 02:32:17.738773 22242 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 198952c9f5644bc5a514d3f6d8a6a871: Bootstrap complete.
I20260812 02:32:17.739478 22242 raft_consensus.cc:348] T 00000000000000000000000000000000 P 198952c9f5644bc5a514d3f6d8a6a871 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "295ba34a46a74756a017dbe2add34dbb" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 37179 } } peers { permanent_uuid: "198952c9f5644bc5a514d3f6d8a6a871" member_type: NON_VOTER last_known_addr { host: "127.14.103.61" port: 35785 } attrs { promote: true } }
I20260812 02:32:17.739643 22242 raft_consensus.cc:729] T 00000000000000000000000000000000 P 198952c9f5644bc5a514d3f6d8a6a871 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 198952c9f5644bc5a514d3f6d8a6a871, State: Initialized, Role: LEARNER
I20260812 02:32:17.739750 22242 consensus_queue.cc:260] T 00000000000000000000000000000000 P 198952c9f5644bc5a514d3f6d8a6a871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "295ba34a46a74756a017dbe2add34dbb" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 37179 } } peers { permanent_uuid: "198952c9f5644bc5a514d3f6d8a6a871" member_type: NON_VOTER last_known_addr { host: "127.14.103.61" port: 35785 } attrs { promote: true } }
I20260812 02:32:17.740125 22246 sys_catalog.cc:455] T 00000000000000000000000000000000 P 198952c9f5644bc5a514d3f6d8a6a871 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "295ba34a46a74756a017dbe2add34dbb" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 37179 } } peers { permanent_uuid: "198952c9f5644bc5a514d3f6d8a6a871" member_type: NON_VOTER last_known_addr { host: "127.14.103.61" port: 35785 } attrs { promote: true } } }
I20260812 02:32:17.740242 22242 sys_catalog.cc:564] T 00000000000000000000000000000000 P 198952c9f5644bc5a514d3f6d8a6a871 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:32:17.740329 22246 sys_catalog.cc:458] T 00000000000000000000000000000000 P 198952c9f5644bc5a514d3f6d8a6a871 [sys.catalog]: This master's current role is: LEARNER
I20260812 02:32:17.744530 22259 catalog_manager.cc:1259] Loaded cluster ID: 6e4d29f3e17c4d179f9de01bd0f0be77
I20260812 02:32:17.744613 22259 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 198952c9f5644bc5a514d3f6d8a6a871: loading cluster ID for follower catalog manager: success
I20260812 02:32:17.746775 22259 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 198952c9f5644bc5a514d3f6d8a6a871: acquiring CA information for follower catalog manager: success
I20260812 02:32:17.748392 22259 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 198952c9f5644bc5a514d3f6d8a6a871: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 02:32:17.752367 21738 heartbeater.cc:503] Master 127.14.103.62:37179 was elected leader, sending a full tablet report...
W20260812 02:32:17.805472 22161 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:32:17.806099 22161 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:32:17.806478 22161 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:32:17.811972 22161 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:32:17.812108 22161 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:32:17.812145 22161 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:32:17.812176 22161 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:32:17.812235 22161 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 02:32:17.818944 22161 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.103.20:46127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.14.103.62:37179,127.14.103.61:35785,127.14.103.60:40443
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/krb5kdc/oryx_127.14.103.60.keytab
--principal=oryx/127.14.103.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.103.60:40443
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.14.103.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22161
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:32:17.820573 22161 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:32:17.821758 22161 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:32:17.833428 22265 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:32:17.833465 22266 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:32:17.833428 22268 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:32:17.833515 22161 server_base.cc:1034] running on GCE node
I20260812 02:32:17.834549 22161 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:32:17.836362 22161 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:32:17.837641 22161 hybrid_clock.cc:648] HybridClock initialized: now 1786501937837605 us; error 62 us; skew 500 ppm
Aug 12 02:32:17 dist-test-slave-2w3w krb5kdc[21470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501937, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.103.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:32:17.843248 22161 init.cc:377] Logged in from keytab as oryx/127.14.103.60@KRBTEST.COM (short username oryx)
I20260812 02:32:17.844468 22161 webserver.cc:460] Webserver started at http://127.14.103.60:39745/ using document root <none> and password file <none>
I20260812 02:32:17.845161 22161 fs_manager.cc:359] Metadata directory not provided
I20260812 02:32:17.845270 22161 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:32:17.845544 22161 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:32:17.848063 22161 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/minicluster-data/master-2/data/instance:
uuid: "c42e0c000a9b4b53a34d41a257df494d"
format_stamp: "Formatted at 2026-08-12 02:32:17 on dist-test-slave-2w3w"
server_key: "3c13a3fa14b3bd5e724bd15bb53dc66b"
server_key_iv: "5c9a06965c7ee4d9e78839280312d6f7"
server_key_version: "encryptionkey@0"
I20260812 02:32:17.848829 22161 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/minicluster-data/master-2/wal/instance:
uuid: "c42e0c000a9b4b53a34d41a257df494d"
format_stamp: "Formatted at 2026-08-12 02:32:17 on dist-test-slave-2w3w"
server_key: "3c13a3fa14b3bd5e724bd15bb53dc66b"
server_key_iv: "5c9a06965c7ee4d9e78839280312d6f7"
server_key_version: "encryptionkey@0"
I20260812 02:32:17.854575 22161 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260812 02:32:17.858311 22275 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:32:17.860641 22161 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260812 02:32:17.860939 22161 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/minicluster-data/master-2/data,/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/minicluster-data/master-2/wal
uuid: "c42e0c000a9b4b53a34d41a257df494d"
format_stamp: "Formatted at 2026-08-12 02:32:17 on dist-test-slave-2w3w"
server_key: "3c13a3fa14b3bd5e724bd15bb53dc66b"
server_key_iv: "5c9a06965c7ee4d9e78839280312d6f7"
server_key_version: "encryptionkey@0"
I20260812 02:32:17.861121 22161 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:32:17.877012 22161 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:32:17.882802 22161 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:32:17.883097 22161 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:32:17.896101 22161 rpc_server.cc:307] RPC server started. Bound to: 127.14.103.60:40443
I20260812 02:32:17.896493 22336 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.103.60:40443 every 8 connection(s)
I20260812 02:32:17.897708 22161 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/minicluster-data/master-2/data/info.pb
I20260812 02:32:17.901043 22337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:32:17.902443 14748 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbep4M4/build/debug/bin/kudu as pid 22161
I20260812 02:32:17.907434 22337 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.103.62" port: 37179 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 02:32:17 dist-test-slave-2w3w krb5kdc[21470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501936, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.14.103.60@KRBTEST.COM
Aug 12 02:32:17 dist-test-slave-2w3w krb5kdc[21470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501937, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.103.60@KRBTEST.COM for oryx/127.14.103.62@KRBTEST.COM
I20260812 02:32:17.927623 22341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:17.904509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43216 (local address 127.14.103.60:40443)
0812 02:32:17.905046 (+ 537us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:17.905061 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:17.905099 (+ 38us) server_negotiation.cc:407] Connection header received
0812 02:32:17.906004 (+ 905us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:17.906025 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:17.906408 (+ 383us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:17.906736 (+ 328us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:32:17.908098 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:17.909165 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:17.910310 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:17.911257 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:17.911435 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:17.915520 (+ 4085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:32:17.915530 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:32:17.915548 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:32:17.915594 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:32:17.919092 (+ 3498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:17.921568 (+ 2476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:17.921579 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:17.921589 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:17.921818 (+ 229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:17.923610 (+ 1792us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:17.923613 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:17.923615 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:17.924246 (+ 631us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:32:17.924438 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0812 02:32:17.926207 (+ 1769us) server_negotiation.cc:299] Negotiation successful
0812 02:32:17.926602 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":358,"spinlock_wait_cycles":1408,"thread_start_us":196,"threads_started":1}
I20260812 02:32:17.929132 21567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:17.909973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54694 (local address 127.14.103.62:37179)
0812 02:32:17.910141 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:17.910146 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:17.910521 (+ 375us) server_negotiation.cc:407] Connection header received
0812 02:32:17.910709 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:17.910715 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:17.910766 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:17.910859 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:32:17.911881 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:17.913099 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:17.914351 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:17.914674 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:17.914808 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:17.919657 (+ 4849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:32:17.919661 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:32:17.919664 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:32:17.919707 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:32:17.925887 (+ 6180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:17.926771 (+ 884us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:17.926777 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:17.926779 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:17.926927 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:17.927406 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:17.927410 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:17.927412 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:17.927965 (+ 553us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:32:17.928229 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0812 02:32:17.928861 (+ 632us) server_negotiation.cc:299] Negotiation successful
0812 02:32:17.929010 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260812 02:32:17.929212 22342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:17.909815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:37179 (local address 127.0.0.1:54694)
0812 02:32:17.910390 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:17.910436 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0812 02:32:17.910597 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:32:17.910922 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:32:17.910929 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:32:17.911446 (+ 517us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:32:17.911620 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:17.911629 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:17.913243 (+ 1614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:17.913249 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:17.914207 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:17.914218 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:17.915069 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:17.915081 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:17.915237 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:17.916721 (+ 1484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:32:17.916735 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:32:17.919092 (+ 2357us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:32:17.926053 (+ 6961us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:32:17.926061 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:32:17.926069 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:32:17.926414 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:32:17.927187 (+ 773us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:32:17.927192 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:32:17.927200 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:32:17.927296 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:32:17.928196 (+ 900us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:32:17.928208 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:32:17.928520 (+ 312us) client_negotiation.cc:769] Sending connection context
0812 02:32:17.928765 (+ 245us) client_negotiation.cc:240] Negotiation successful
0812 02:32:17.929016 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":373,"spinlock_wait_cycles":2944,"thread_start_us":243,"threads_started":1}
I20260812 02:32:17.931524 22337 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.103.61" port: 35785 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 02:32:17 dist-test-slave-2w3w krb5kdc[21470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501937, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.103.60@KRBTEST.COM for oryx/127.14.103.61@KRBTEST.COM
I20260812 02:32:17.948577 22342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:17.932183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.61:35785 (local address 127.0.0.1:34262)
0812 02:32:17.932507 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:17.932524 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 02:32:17.932623 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:32:17.933369 (+ 746us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:32:17.933371 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:32:17.933552 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:32:17.933885 (+ 333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:17.933893 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:17.935182 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:17.935186 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:17.935845 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:17.935860 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:17.936872 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:17.936877 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:17.936993 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:17.938275 (+ 1282us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:32:17.938294 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:32:17.943552 (+ 5258us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:32:17.946317 (+ 2765us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:32:17.946320 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:32:17.946322 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:32:17.946557 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:32:17.947018 (+ 461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:32:17.947025 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:32:17.947030 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:32:17.947130 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:32:17.947970 (+ 840us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:32:17.947973 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:32:17.948077 (+ 104us) client_negotiation.cc:769] Sending connection context
0812 02:32:17.948178 (+ 101us) client_negotiation.cc:240] Negotiation successful
0812 02:32:17.948435 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216}
I20260812 02:32:17.948612 22343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:17.932525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34262 (local address 127.14.103.61:35785)
0812 02:32:17.932984 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:17.932989 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:17.933013 (+ 24us) server_negotiation.cc:407] Connection header received
0812 02:32:17.933076 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:17.933080 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:17.933145 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:17.933293 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:32:17.934100 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:17.934825 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:17.936062 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:17.936335 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:17.936503 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:17.944132 (+ 7629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:32:17.944136 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:32:17.944139 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:32:17.944174 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:32:17.946067 (+ 1893us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:17.946689 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:17.946692 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:17.946693 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:17.946782 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:17.947482 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:17.947487 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:17.947489 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:17.947708 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:32:17.947940 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0812 02:32:17.948323 (+ 383us) server_negotiation.cc:299] Negotiation successful
0812 02:32:17.948462 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":131,"threads_started":1}
I20260812 02:32:17.949926 22337 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.103.60" port: 40443 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 02:32:17 dist-test-slave-2w3w krb5kdc[21470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501937, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.103.60@KRBTEST.COM for oryx/127.14.103.60@KRBTEST.COM
I20260812 02:32:17.967722 22342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:17.950450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.60:40443 (local address 127.0.0.1:43232)
0812 02:32:17.950637 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:17.950654 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 02:32:17.950754 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:32:17.951067 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:32:17.951069 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:32:17.951286 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:32:17.951356 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:17.951362 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:17.952735 (+ 1373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:17.952738 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:17.953318 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:17.953327 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:17.953787 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:17.953790 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:17.953857 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:17.954478 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:32:17.954492 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:32:17.956165 (+ 1673us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:32:17.958035 (+ 1870us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:32:17.958039 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:32:17.958042 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:32:17.958246 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:32:17.958542 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:32:17.958547 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:32:17.958549 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:32:17.958872 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:32:17.965057 (+ 6185us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:32:17.965061 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:32:17.965547 (+ 486us) client_negotiation.cc:769] Sending connection context
0812 02:32:17.966018 (+ 471us) client_negotiation.cc:240] Negotiation successful
0812 02:32:17.966738 (+ 720us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87}
I20260812 02:32:17.967916 22341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:17.950584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43232 (local address 127.14.103.60:40443)
0812 02:32:17.950692 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:17.950699 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:17.950727 (+ 28us) server_negotiation.cc:407] Connection header received
0812 02:32:17.950867 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:17.950870 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:17.950916 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:17.951044 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:32:17.951484 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:17.952500 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:17.953451 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:17.953670 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:17.953783 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:17.956347 (+ 2564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:32:17.956351 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:32:17.956353 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:32:17.956378 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:32:17.957896 (+ 1518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:17.958352 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:17.958358 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:17.958360 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:17.958440 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:17.963993 (+ 5553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:17.964003 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:17.964005 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:17.964597 (+ 592us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:32:17.965204 (+ 607us) server_negotiation.cc:1035] Waiting for connection context
0812 02:32:17.966313 (+ 1109us) server_negotiation.cc:299] Negotiation successful
0812 02:32:17.967654 (+ 1341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":225,"server-negotiator.queue_time_us":45}
I20260812 02:32:17.971989 22337 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c42e0c000a9b4b53a34d41a257df494d: Bootstrap starting.
I20260812 02:32:17.974944 22337 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c42e0c000a9b4b53a34d41a257df494d: Neither blocks nor log segments found. Creating new log.
I20260812 02:32:17.976109 22337 log.cc:824] T 00000000000000000000000000000000 P c42e0c000a9b4b53a34d41a257df494d: Log is configured to *not* fsync() on all Append() calls
I20260812 02:32:17.978899 22337 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c42e0c000a9b4b53a34d41a257df494d: No bootstrap required, opened a new log
I20260812 02:32:17.991204 22337 raft_consensus.cc:348] T 00000000000000000000000000000000 P c42e0c000a9b4b53a34d41a257df494d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "295ba34a46a74756a017dbe2add34dbb" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 37179 } } peers { permanent_uuid: "198952c9f5644bc5a514d3f6d8a6a871" member_type: VOTER last_known_addr { host: "127.14.103.61" port: 35785 } } peers { permanent_uuid: "c42e0c000a9b4b53a34d41a257df494d" member_type: VOTER last_known_addr { host: "127.14.103.60" port: 40443 } }
I20260812 02:32:17.991767 22337 raft_consensus.cc:374] T 00000000000000000000000000000000 P c42e0c000a9b4b53a34d41a257df494d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:32:17.991866 22337 raft_consensus.cc:729] T 00000000000000000000000000000000 P c42e0c000a9b4b53a34d41a257df494d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c42e0c000a9b4b53a34d41a257df494d, State: Initialized, Role: FOLLOWER
I20260812 02:32:17.992374 22337 consensus_queue.cc:260] T 00000000000000000000000000000000 P c42e0c000a9b4b53a34d41a257df494d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "295ba34a46a74756a017dbe2add34dbb" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 37179 } } peers { permanent_uuid: "198952c9f5644bc5a514d3f6d8a6a871" member_type: VOTER last_known_addr { host: "127.14.103.61" port: 35785 } } peers { permanent_uuid: "c42e0c000a9b4b53a34d41a257df494d" member_type: VOTER last_known_addr { host: "127.14.103.60" port: 40443 } }
I20260812 02:32:17.993124 22345 sys_catalog.cc:455] T 00000000000000000000000000000000 P c42e0c000a9b4b53a34d41a257df494d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "295ba34a46a74756a017dbe2add34dbb" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 37179 } } peers { permanent_uuid: "198952c9f5644bc5a514d3f6d8a6a871" member_type: VOTER last_known_addr { host: "127.14.103.61" port: 35785 } } peers { permanent_uuid: "c42e0c000a9b4b53a34d41a257df494d" member_type: VOTER last_known_addr { host: "127.14.103.60" port: 40443 } } }
I20260812 02:32:17.993258 22345 sys_catalog.cc:458] T 00000000000000000000000000000000 P c42e0c000a9b4b53a34d41a257df494d [sys.catalog]: This master's current role is: FOLLOWER
I20260812 02:32:17.993781 22337 sys_catalog.cc:564] T 00000000000000000000000000000000 P c42e0c000a9b4b53a34d41a257df494d [sys.catalog]: configured and running, proceeding with master startup.
W20260812 02:32:18.000048 22358 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c42e0c000a9b4b53a34d41a257df494d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 02:32:18.000190 22358 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 02:32:18.009196 22161 master_runner.cc:428] Detected that this master c42e0c000a9b4b53a34d41a257df494d is joining an existing cluster
I20260812 02:32:18.009302 22161 master_runner.cc:432] Initiating AddMaster RPC to add 127.14.103.60:40443
I20260812 02:32:18.015753 21885 heartbeater.cc:503] Master 127.14.103.62:37179 was elected leader, sending a full tablet report...
I20260812 02:32:18.037647 22341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:18.012999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43240 (local address 127.14.103.60:40443)
0812 02:32:18.013101 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:18.013105 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:18.013855 (+ 750us) server_negotiation.cc:407] Connection header received
0812 02:32:18.014011 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:18.014014 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:18.014057 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:18.014107 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:32:18.014672 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:18.015446 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:18.017911 (+ 2465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:18.018141 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:18.018258 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:18.019880 (+ 1622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:32:18.019884 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:32:18.019887 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:32:18.019916 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:32:18.021997 (+ 2081us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:18.036105 (+ 14108us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:18.036109 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:18.036111 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:18.036226 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:18.036598 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:18.036602 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:18.036603 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:18.036905 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:32:18.037000 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0812 02:32:18.037329 (+ 329us) server_negotiation.cc:299] Negotiation successful
0812 02:32:18.037473 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260812 02:32:18.038081 14748 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbep4M4/build/debug/bin/kudu with pid 21592
I20260812 02:32:18.038177 22367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:18.012907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.60:40443 (local address 127.0.0.1:43240)
0812 02:32:18.013773 (+ 866us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:18.013788 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 02:32:18.013907 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:32:18.014230 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:32:18.014234 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:32:18.014442 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:32:18.014559 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:18.014565 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:18.016498 (+ 1933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:18.016501 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:18.017777 (+ 1276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:18.017797 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:18.018274 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:18.018277 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:18.018384 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:18.019073 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:32:18.019090 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:32:18.019660 (+ 570us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:32:18.022121 (+ 2461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:32:18.022127 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:32:18.022130 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:32:18.035901 (+ 13771us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:32:18.036347 (+ 446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:32:18.036350 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:32:18.036352 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:32:18.036482 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:32:18.037067 (+ 585us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:32:18.037070 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:32:18.037171 (+ 101us) client_negotiation.cc:769] Sending connection context
0812 02:32:18.037859 (+ 688us) client_negotiation.cc:240] Negotiation successful
0812 02:32:18.038012 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":755,"thread_start_us":115,"threads_started":1}
I20260812 02:32:18.043165 22364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:18.012291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.61:35785 (local address 127.0.0.1:34278)
0812 02:32:18.013250 (+ 959us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:18.013261 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 02:32:18.013375 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:32:18.013826 (+ 451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:32:18.013831 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:32:18.014148 (+ 317us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:32:18.014267 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:18.014276 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:18.015429 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:18.015432 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:18.016275 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:18.016286 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:18.017203 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:18.017207 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:18.017376 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:18.018411 (+ 1035us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:32:18.018424 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:32:18.035901 (+ 17477us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:32:18.039956 (+ 4055us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:32:18.039964 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:32:18.039969 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:32:18.041157 (+ 1188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:32:18.041718 (+ 561us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:32:18.041725 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:32:18.041730 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:32:18.041844 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:32:18.042380 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:32:18.042385 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:32:18.042519 (+ 134us) client_negotiation.cc:769] Sending connection context
0812 02:32:18.042623 (+ 104us) client_negotiation.cc:240] Negotiation successful
0812 02:32:18.042814 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":907,"mutex_wait_us":42,"thread_start_us":125,"threads_started":1}
I20260812 02:32:18.045217 22343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:18.012398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34278 (local address 127.14.103.61:35785)
0812 02:32:18.012542 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:18.012547 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:18.013346 (+ 799us) server_negotiation.cc:407] Connection header received
0812 02:32:18.013497 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:18.013502 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:18.013575 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:18.013676 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:32:18.014413 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:18.015298 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:18.016575 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:18.016933 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:18.017050 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:18.036210 (+ 19160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:32:18.036215 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:32:18.036218 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:32:18.036259 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:32:18.038942 (+ 2683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:18.041418 (+ 2476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:18.041426 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:18.041431 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:18.041588 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:18.041952 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:18.041956 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:18.041958 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:18.042236 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:32:18.042336 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0812 02:32:18.043870 (+ 1534us) server_negotiation.cc:299] Negotiation successful
0812 02:32:18.044083 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260812 02:32:18.051708 22365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:18.012291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:37179 (local address 127.0.0.1:54704)
0812 02:32:18.012867 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:18.012885 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 02:32:18.013001 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:32:18.013500 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:32:18.013504 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:32:18.013798 (+ 294us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:32:18.013986 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:18.013995 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:18.015226 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:18.015229 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:18.018046 (+ 2817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:18.018083 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:18.019380 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:18.019384 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:18.019482 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:18.019967 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:32:18.019983 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:32:18.036139 (+ 16156us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:32:18.049735 (+ 13596us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:32:18.049739 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:32:18.049742 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:32:18.050061 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:32:18.050420 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:32:18.050424 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:32:18.050426 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:32:18.050512 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:32:18.051066 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:32:18.051069 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:32:18.051198 (+ 129us) client_negotiation.cc:769] Sending connection context
0812 02:32:18.051320 (+ 122us) client_negotiation.cc:240] Negotiation successful
0812 02:32:18.051500 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":148,"threads_started":1}
I20260812 02:32:18.051954 21567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:18.012537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54704 (local address 127.14.103.62:37179)
0812 02:32:18.012661 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:18.012665 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:18.013135 (+ 470us) server_negotiation.cc:407] Connection header received
0812 02:32:18.013195 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:18.013198 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:18.013252 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:18.013350 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:32:18.014140 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:18.015071 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:18.018289 (+ 3218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:18.019236 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:18.019364 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:18.036348 (+ 16984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:32:18.036355 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:32:18.036361 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:32:18.036399 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:32:18.049238 (+ 12839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:18.050199 (+ 961us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:18.050203 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:18.050205 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:18.050308 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:18.050617 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:18.050620 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:18.050623 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:18.050871 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:32:18.050952 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0812 02:32:18.051435 (+ 483us) server_negotiation.cc:299] Negotiation successful
0812 02:32:18.051626 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260812 02:32:18.055191 14748 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbep4M4/build/debug/bin/kudu
/tmp/dist-test-taskbep4M4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.103.1:46041
--local_ip_for_outbound_sockets=127.14.103.1
--tserver_master_addrs=127.14.103.62:37179
--webserver_port=41597
--webserver_interface=127.14.103.1
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/krb5kdc/oryx_127.14.103.1.keytab
--principal=oryx/127.14.103.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.103.20:46127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.14.103.62:37179,127.14.103.61:35785
--tserver_master_addrs=127.14.103.62:37179,127.14.103.61:35785,127.14.103.60:40443 with env {KRB5CCNAME=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 02:32:18 dist-test-slave-2w3w krb5kdc[21470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501936, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.103.62@KRBTEST.COM for oryx/127.14.103.60@KRBTEST.COM
I20260812 02:32:18.078269 22152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:18.057620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.60:40443 (local address 127.0.0.1:43246)
0812 02:32:18.057831 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:18.057857 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0812 02:32:18.057957 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:32:18.058401 (+ 444us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:32:18.058405 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:32:18.058657 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:32:18.058789 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:18.058796 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:18.060754 (+ 1958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:18.060763 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:18.063408 (+ 2645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:18.063454 (+ 46us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:18.065735 (+ 2281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:18.065741 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:18.065918 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:18.070949 (+ 5031us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:32:18.070999 (+ 50us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:32:18.073311 (+ 2312us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:32:18.076369 (+ 3058us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:32:18.076374 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:32:18.076377 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:32:18.076702 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:32:18.077043 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:32:18.077046 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:32:18.077049 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:32:18.077154 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:32:18.077626 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:32:18.077629 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:32:18.077757 (+ 128us) client_negotiation.cc:769] Sending connection context
0812 02:32:18.077871 (+ 114us) client_negotiation.cc:240] Negotiation successful
0812 02:32:18.078078 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260812 02:32:18.078267 22341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:18.057861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43246 (local address 127.14.103.60:40443)
0812 02:32:18.058017 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:18.058025 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:18.058043 (+ 18us) server_negotiation.cc:407] Connection header received
0812 02:32:18.058152 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:18.058158 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:18.058240 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:18.058375 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:32:18.058937 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:18.060066 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:18.064209 (+ 4143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:18.065210 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:18.065432 (+ 222us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:18.073853 (+ 8421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:32:18.073859 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:32:18.073862 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:32:18.073900 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:32:18.076175 (+ 2275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:18.076833 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:18.076837 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:18.076838 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:18.076937 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:18.077262 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:18.077265 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:18.077267 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:18.077521 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:32:18.077624 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0812 02:32:18.077959 (+ 335us) server_negotiation.cc:299] Negotiation successful
0812 02:32:18.078097 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260812 02:32:18.080526 21520 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.14.103.60:40443: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "c42e0c000a9b4b53a34d41a257df494d"
member_type: NON_VOTER
last_known_addr {
host: "127.14.103.60"
port: 40443
}
attrs {
promote: true
}
}
dest_uuid: "295ba34a46a74756a017dbe2add34dbb"
cas_config_opid_index: 5
I20260812 02:32:18.081779 21520 consensus_queue.cc:237] T 00000000000000000000000000000000 P 295ba34a46a74756a017dbe2add34dbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "295ba34a46a74756a017dbe2add34dbb" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 37179 } } peers { permanent_uuid: "198952c9f5644bc5a514d3f6d8a6a871" member_type: NON_VOTER last_known_addr { host: "127.14.103.61" port: 35785 } attrs { promote: true } } peers { permanent_uuid: "c42e0c000a9b4b53a34d41a257df494d" member_type: NON_VOTER last_known_addr { host: "127.14.103.60" port: 40443 } attrs { promote: true } }
I20260812 02:32:18.085920 22155 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 295ba34a46a74756a017dbe2add34dbb [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER c42e0c000a9b4b53a34d41a257df494d (127.14.103.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "295ba34a46a74756a017dbe2add34dbb" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 37179 } } peers { permanent_uuid: "198952c9f5644bc5a514d3f6d8a6a871" member_type: NON_VOTER last_known_addr { host: "127.14.103.61" port: 35785 } attrs { promote: true } } peers { permanent_uuid: "c42e0c000a9b4b53a34d41a257df494d" member_type: NON_VOTER last_known_addr { host: "127.14.103.60" port: 40443 } attrs { promote: true } } }
I20260812 02:32:18.088708 22155 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.14.103.60:40443
I20260812 02:32:18.088943 22155 sys_catalog.cc:455] T 00000000000000000000000000000000 P 295ba34a46a74756a017dbe2add34dbb [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "295ba34a46a74756a017dbe2add34dbb" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "295ba34a46a74756a017dbe2add34dbb" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 37179 } } peers { permanent_uuid: "198952c9f5644bc5a514d3f6d8a6a871" member_type: NON_VOTER last_known_addr { host: "127.14.103.61" port: 35785 } attrs { promote: true } } peers { permanent_uuid: "c42e0c000a9b4b53a34d41a257df494d" member_type: NON_VOTER last_known_addr { host: "127.14.103.60" port: 40443 } attrs { promote: true } } }
I20260812 02:32:18.089058 22155 sys_catalog.cc:458] T 00000000000000000000000000000000 P 295ba34a46a74756a017dbe2add34dbb [sys.catalog]: This master's current role is: LEADER
I20260812 02:32:18.090631 22161 master.cc:537] Master@127.14.103.60:40443 shutting down...
W20260812 02:32:18.092890 21502 consensus_peers.cc:487] T 00000000000000000000000000000000 P 295ba34a46a74756a017dbe2add34dbb -> Peer c42e0c000a9b4b53a34d41a257df494d (127.14.103.60:40443): Couldn't send request to peer c42e0c000a9b4b53a34d41a257df494d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260812 02:32:18.096678 22152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:18.089803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.61:35785 (local address 127.0.0.1:34292)
0812 02:32:18.090082 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:18.090101 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 02:32:18.090214 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:32:18.091564 (+ 1350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:32:18.091571 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:32:18.091597 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:32:18.091723 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:18.091730 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:18.092925 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:18.092929 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:18.094331 (+ 1402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:18.094348 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:18.096102 (+ 1754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:18.096107 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:18.096224 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:18.096299 (+ 75us) client_negotiation.cc:769] Sending connection context
0812 02:32:18.096390 (+ 91us) client_negotiation.cc:240] Negotiation successful
0812 02:32:18.096506 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178}
I20260812 02:32:18.097505 22343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:18.089926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34292 (local address 127.14.103.61:35785)
0812 02:32:18.090647 (+ 721us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:18.090651 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:18.090666 (+ 15us) server_negotiation.cc:407] Connection header received
0812 02:32:18.090720 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:18.090724 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:18.090791 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:18.090906 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:32:18.091965 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:18.092765 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:18.094675 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:18.095491 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:18.095623 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:18.095791 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0812 02:32:18.097255 (+ 1464us) server_negotiation.cc:299] Negotiation successful
0812 02:32:18.097358 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":631}
I20260812 02:32:18.098150 22213 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 198952c9f5644bc5a514d3f6d8a6a871 [term 1 LEARNER]: Refusing update from remote peer 295ba34a46a74756a017dbe2add34dbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260812 02:32:18.099501 22369 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 295ba34a46a74756a017dbe2add34dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "198952c9f5644bc5a514d3f6d8a6a871" member_type: NON_VOTER last_known_addr { host: "127.14.103.61" port: 35785 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260812 02:32:18.103929 22213 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 198952c9f5644bc5a514d3f6d8a6a871 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER c42e0c000a9b4b53a34d41a257df494d (127.14.103.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "295ba34a46a74756a017dbe2add34dbb" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 37179 } } peers { permanent_uuid: "198952c9f5644bc5a514d3f6d8a6a871" member_type: NON_VOTER last_known_addr { host: "127.14.103.61" port: 35785 } attrs { promote: true } } peers { permanent_uuid: "c42e0c000a9b4b53a34d41a257df494d" member_type: NON_VOTER last_known_addr { host: "127.14.103.60" port: 40443 } attrs { promote: true } } }
I20260812 02:32:18.104836 22161 raft_consensus.cc:2229] T 00000000000000000000000000000000 P c42e0c000a9b4b53a34d41a257df494d [term 0 FOLLOWER]: Raft consensus shutting down.
I20260812 02:32:18.105157 22161 raft_consensus.cc:2258] T 00000000000000000000000000000000 P c42e0c000a9b4b53a34d41a257df494d [term 0 FOLLOWER]: Raft consensus is shut down!
I20260812 02:32:18.105295 22161 tablet_replica.cc:330] T 00000000000000000000000000000000 P c42e0c000a9b4b53a34d41a257df494d: stopping tablet replica
I20260812 02:32:18.108456 22246 sys_catalog.cc:455] T 00000000000000000000000000000000 P 198952c9f5644bc5a514d3f6d8a6a871 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 295ba34a46a74756a017dbe2add34dbb. Latest consensus state: current_term: 1 leader_uuid: "295ba34a46a74756a017dbe2add34dbb" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "295ba34a46a74756a017dbe2add34dbb" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 37179 } } peers { permanent_uuid: "198952c9f5644bc5a514d3f6d8a6a871" member_type: NON_VOTER last_known_addr { host: "127.14.103.61" port: 35785 } attrs { promote: true } } peers { permanent_uuid: "c42e0c000a9b4b53a34d41a257df494d" member_type: NON_VOTER last_known_addr { host: "127.14.103.60" port: 40443 } attrs { promote: true } } }
I20260812 02:32:18.108652 22246 sys_catalog.cc:458] T 00000000000000000000000000000000 P 198952c9f5644bc5a514d3f6d8a6a871 [sys.catalog]: This master's current role is: LEARNER
I20260812 02:32:18.108784 22246 sys_catalog.cc:455] T 00000000000000000000000000000000 P 198952c9f5644bc5a514d3f6d8a6a871 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "295ba34a46a74756a017dbe2add34dbb" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "295ba34a46a74756a017dbe2add34dbb" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 37179 } } peers { permanent_uuid: "198952c9f5644bc5a514d3f6d8a6a871" member_type: NON_VOTER last_known_addr { host: "127.14.103.61" port: 35785 } attrs { promote: true } } peers { permanent_uuid: "c42e0c000a9b4b53a34d41a257df494d" member_type: NON_VOTER last_known_addr { host: "127.14.103.60" port: 40443 } attrs { promote: true } } }
I20260812 02:32:18.108888 22246 sys_catalog.cc:458] T 00000000000000000000000000000000 P 198952c9f5644bc5a514d3f6d8a6a871 [sys.catalog]: This master's current role is: LEARNER
I20260812 02:32:18.110201 22369 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 295ba34a46a74756a017dbe2add34dbb: attempting to promote NON_VOTER 198952c9f5644bc5a514d3f6d8a6a871 to VOTER
I20260812 02:32:18.110791 22369 consensus_queue.cc:237] T 00000000000000000000000000000000 P 295ba34a46a74756a017dbe2add34dbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "295ba34a46a74756a017dbe2add34dbb" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 37179 } } peers { permanent_uuid: "198952c9f5644bc5a514d3f6d8a6a871" member_type: VOTER last_known_addr { host: "127.14.103.61" port: 35785 } attrs { promote: false } } peers { permanent_uuid: "c42e0c000a9b4b53a34d41a257df494d" member_type: NON_VOTER last_known_addr { host: "127.14.103.60" port: 40443 } attrs { promote: true } }
I20260812 02:32:18.112246 22152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:18.111878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.60:40443 (local address 127.0.0.1:43256)
0812 02:32:18.112072 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:18.112154 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.60:40443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114}
I20260812 02:32:18.113131 22213 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 198952c9f5644bc5a514d3f6d8a6a871 [term 1 LEARNER]: Refusing update from remote peer 295ba34a46a74756a017dbe2add34dbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260812 02:32:18.113345 22152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:18.113037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.60:40443 (local address 127.0.0.1:43262)
0812 02:32:18.113206 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:18.113257 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.60:40443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
W20260812 02:32:18.113622 21502 consensus_peers.cc:487] T 00000000000000000000000000000000 P 295ba34a46a74756a017dbe2add34dbb -> Peer c42e0c000a9b4b53a34d41a257df494d (127.14.103.60:40443): Couldn't send request to peer c42e0c000a9b4b53a34d41a257df494d. Status: Network error: Client connection negotiation failed: client connection to 127.14.103.60:40443: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260812 02:32:18.113677 22155 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 295ba34a46a74756a017dbe2add34dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "198952c9f5644bc5a514d3f6d8a6a871" member_type: VOTER last_known_addr { host: "127.14.103.61" port: 35785 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260812 02:32:18.114866 22161 master.cc:552] Master@127.14.103.60:40443 shutdown complete.
I20260812 02:32:18.114924 22161 master_runner.cc:305] Clearing existing system tablet
I20260812 02:32:18.115439 22161 fs_manager.cc:359] Metadata directory not provided
I20260812 02:32:18.115512 22161 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:32:18.116900 22161 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260812 02:32:18.117180 22155 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 295ba34a46a74756a017dbe2add34dbb [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 198952c9f5644bc5a514d3f6d8a6a871 (127.14.103.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "295ba34a46a74756a017dbe2add34dbb" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 37179 } } peers { permanent_uuid: "198952c9f5644bc5a514d3f6d8a6a871" member_type: VOTER last_known_addr { host: "127.14.103.61" port: 35785 } attrs { promote: false } } peers { permanent_uuid: "c42e0c000a9b4b53a34d41a257df494d" member_type: NON_VOTER last_known_addr { host: "127.14.103.60" port: 40443 } attrs { promote: true } } }
I20260812 02:32:18.118520 22374 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:32:18.118746 22161 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:32:18.118861 22161 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 02:32:18.118894 22161 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/minicluster-data/master-2/data,/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/minicluster-data/master-2/wal
uuid: "c42e0c000a9b4b53a34d41a257df494d"
format_stamp: "Formatted at 2026-08-12 02:32:17 on dist-test-slave-2w3w"
server_key: "3c13a3fa14b3bd5e724bd15bb53dc66b"
server_key_iv: "5c9a06965c7ee4d9e78839280312d6f7"
server_key_version: "encryptionkey@0"
I20260812 02:32:18.119063 22155 sys_catalog.cc:455] T 00000000000000000000000000000000 P 295ba34a46a74756a017dbe2add34dbb [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "295ba34a46a74756a017dbe2add34dbb" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "295ba34a46a74756a017dbe2add34dbb" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 37179 } } peers { permanent_uuid: "198952c9f5644bc5a514d3f6d8a6a871" member_type: VOTER last_known_addr { host: "127.14.103.61" port: 35785 } attrs { promote: false } } peers { permanent_uuid: "c42e0c000a9b4b53a34d41a257df494d" member_type: NON_VOTER last_known_addr { host: "127.14.103.60" port: 40443 } attrs { promote: true } } }
I20260812 02:32:18.119187 22155 sys_catalog.cc:458] T 00000000000000000000000000000000 P 295ba34a46a74756a017dbe2add34dbb [sys.catalog]: This master's current role is: LEADER
I20260812 02:32:18.119307 22155 sys_catalog.cc:455] T 00000000000000000000000000000000 P 295ba34a46a74756a017dbe2add34dbb [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "295ba34a46a74756a017dbe2add34dbb" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "295ba34a46a74756a017dbe2add34dbb" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 37179 } } peers { permanent_uuid: "198952c9f5644bc5a514d3f6d8a6a871" member_type: VOTER last_known_addr { host: "127.14.103.61" port: 35785 } attrs { promote: false } } peers { permanent_uuid: "c42e0c000a9b4b53a34d41a257df494d" member_type: NON_VOTER last_known_addr { host: "127.14.103.60" port: 40443 } attrs { promote: true } } }
I20260812 02:32:18.119374 22155 sys_catalog.cc:458] T 00000000000000000000000000000000 P 295ba34a46a74756a017dbe2add34dbb [sys.catalog]: This master's current role is: LEADER
I20260812 02:32:18.120549 22213 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 198952c9f5644bc5a514d3f6d8a6a871 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 198952c9f5644bc5a514d3f6d8a6a871 (127.14.103.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "295ba34a46a74756a017dbe2add34dbb" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 37179 } } peers { permanent_uuid: "198952c9f5644bc5a514d3f6d8a6a871" member_type: VOTER last_known_addr { host: "127.14.103.61" port: 35785 } attrs { promote: false } } peers { permanent_uuid: "c42e0c000a9b4b53a34d41a257df494d" member_type: NON_VOTER last_known_addr { host: "127.14.103.60" port: 40443 } attrs { promote: true } } }
I20260812 02:32:18.122084 22161 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P c42e0c000a9b4b53a34d41a257df494d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:32:18.125313 22372 sys_catalog.cc:455] T 00000000000000000000000000000000 P 198952c9f5644bc5a514d3f6d8a6a871 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "295ba34a46a74756a017dbe2add34dbb" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "295ba34a46a74756a017dbe2add34dbb" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 37179 } } peers { permanent_uuid: "198952c9f5644bc5a514d3f6d8a6a871" member_type: VOTER last_known_addr { host: "127.14.103.61" port: 35785 } attrs { promote: false } } peers { permanent_uuid: "c42e0c000a9b4b53a34d41a257df494d" member_type: NON_VOTER last_known_addr { host: "127.14.103.60" port: 40443 } attrs { promote: true } } }
I20260812 02:32:18.125486 22372 sys_catalog.cc:458] T 00000000000000000000000000000000 P 198952c9f5644bc5a514d3f6d8a6a871 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 02:32:18.125877 22161 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P c42e0c000a9b4b53a34d41a257df494d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260812 02:32:18.126320 22161 log.cc:1196] T 00000000000000000000000000000000 P c42e0c000a9b4b53a34d41a257df494d: Deleting WAL directory at /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260812 02:32:18.127909 22161 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P c42e0c000a9b4b53a34d41a257df494d: Deleting consensus metadata
I20260812 02:32:18.128242 22161 master_runner.cc:315] Copying system tablet from 127.14.103.62:37179
I20260812 02:32:18.129868 22161 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P c42e0c000a9b4b53a34d41a257df494d: tablet copy: Beginning tablet copy session from remote peer at address 127.14.103.62:37179
I20260812 02:32:18.146591 21567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:18.130627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54716 (local address 127.14.103.62:37179)
0812 02:32:18.130763 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:18.130766 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:18.130907 (+ 141us) server_negotiation.cc:407] Connection header received
0812 02:32:18.131076 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:18.131079 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:18.131133 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:18.131238 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:32:18.131821 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:18.132642 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:18.133971 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:18.134320 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:18.134468 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:18.136687 (+ 2219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:32:18.136697 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:32:18.136700 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:32:18.136767 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:32:18.139506 (+ 2739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:18.141174 (+ 1668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:18.141179 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:18.141181 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:18.143054 (+ 1873us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:18.144540 (+ 1486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:18.144545 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:18.144547 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:18.144890 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:32:18.145063 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0812 02:32:18.146220 (+ 1157us) server_negotiation.cc:299] Negotiation successful
0812 02:32:18.146410 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260812 02:32:18.147279 22380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:18.130503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:37179 (local address 127.0.0.1:54716)
0812 02:32:18.130802 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:18.130817 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 02:32:18.130982 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:32:18.131281 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:32:18.131284 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:32:18.131540 (+ 256us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:32:18.131664 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:18.131671 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:18.132790 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:18.132794 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:18.133769 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:18.133783 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:18.134490 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:18.134494 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:18.134597 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:18.135376 (+ 779us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:32:18.135392 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:32:18.136362 (+ 970us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:32:18.139815 (+ 3453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:32:18.139820 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:32:18.139823 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:32:18.140972 (+ 1149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:32:18.143731 (+ 2759us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:32:18.143736 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:32:18.143738 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:32:18.144092 (+ 354us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:32:18.145573 (+ 1481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:32:18.145578 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:32:18.145934 (+ 356us) client_negotiation.cc:769] Sending connection context
0812 02:32:18.146099 (+ 165us) client_negotiation.cc:240] Negotiation successful
0812 02:32:18.146468 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":71,"threads_started":1}
I20260812 02:32:18.147830 21545 tablet_copy_service.cc:140] P 295ba34a46a74756a017dbe2add34dbb: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer c42e0c000a9b4b53a34d41a257df494d ({username='oryx', principal='oryx/127.14.103.60@KRBTEST.COM'} at 127.0.0.1:54716)
I20260812 02:32:18.148069 21545 tablet_copy_service.cc:161] P 295ba34a46a74756a017dbe2add34dbb: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer c42e0c000a9b4b53a34d41a257df494d at {username='oryx', principal='oryx/127.14.103.60@KRBTEST.COM'} at 127.0.0.1:54716: session id = c42e0c000a9b4b53a34d41a257df494d-00000000000000000000000000000000
I20260812 02:32:18.150046 21545 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 295ba34a46a74756a017dbe2add34dbb: Tablet Copy: opened 0 blocks and 1 log segments
I20260812 02:32:18.151746 22161 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260812 02:32:18.154875 22161 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P c42e0c000a9b4b53a34d41a257df494d: tablet copy: Starting download of 0 data blocks...
I20260812 02:32:18.155167 22161 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P c42e0c000a9b4b53a34d41a257df494d: tablet copy: Starting download of 1 WAL segments...
I20260812 02:32:18.157199 22161 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P c42e0c000a9b4b53a34d41a257df494d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260812 02:32:18.163023 21545 tablet_copy_service.cc:342] P 295ba34a46a74756a017dbe2add34dbb: Request end of tablet copy session c42e0c000a9b4b53a34d41a257df494d-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.14.103.60@KRBTEST.COM'} at 127.0.0.1:54716
I20260812 02:32:18.163187 21545 tablet_copy_service.cc:433] P 295ba34a46a74756a017dbe2add34dbb: ending tablet copy session c42e0c000a9b4b53a34d41a257df494d-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer c42e0c000a9b4b53a34d41a257df494d
I20260812 02:32:18.173708 22161 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:32:18.175953 22381 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:32:18.175987 22384 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:32:18.176437 22161 server_base.cc:1034] running on GCE node
W20260812 02:32:18.176608 22382 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:32:18.176882 22161 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:32:18.177256 22161 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:32:18.178424 22161 hybrid_clock.cc:648] HybridClock initialized: now 1786501938178398 us; error 49 us; skew 500 ppm
Aug 12 02:32:18 dist-test-slave-2w3w krb5kdc[21470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501938, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.103.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:32:18.181722 22161 init.cc:377] Logged in from keytab as oryx/127.14.103.60@KRBTEST.COM (short username oryx)
I20260812 02:32:18.182960 22161 webserver.cc:460] Webserver started at http://127.14.103.60:42725/ using document root <none> and password file <none>
I20260812 02:32:18.183298 22161 fs_manager.cc:359] Metadata directory not provided
I20260812 02:32:18.183338 22161 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:32:18.184962 22161 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260812 02:32:18.186945 22395 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:32:18.187980 22161 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260812 02:32:18.188151 22161 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/minicluster-data/master-2/data,/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/minicluster-data/master-2/wal
uuid: "c42e0c000a9b4b53a34d41a257df494d"
format_stamp: "Formatted at 2026-08-12 02:32:17 on dist-test-slave-2w3w"
server_key: "3c13a3fa14b3bd5e724bd15bb53dc66b"
server_key_iv: "5c9a06965c7ee4d9e78839280312d6f7"
server_key_version: "encryptionkey@0"
I20260812 02:32:18.188304 22161 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:32:18.220065 22161 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:32:18.223263 22161 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:32:18.229437 22161 rpc_server.cc:307] RPC server started. Bound to: 127.14.103.60:40443
I20260812 02:32:18.229703 22454 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.103.60:40443 every 8 connection(s)
I20260812 02:32:18.230173 22161 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/minicluster-data/master-2/data/info.pb
I20260812 02:32:18.232442 22455 sys_catalog.cc:263] Verifying existing consensus state
I20260812 02:32:18.233677 22455 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c42e0c000a9b4b53a34d41a257df494d: Bootstrap starting.
I20260812 02:32:18.250823 22455 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c42e0c000a9b4b53a34d41a257df494d: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 02:32:18.251379 22455 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c42e0c000a9b4b53a34d41a257df494d: Bootstrap complete.
I20260812 02:32:18.252146 22455 raft_consensus.cc:348] T 00000000000000000000000000000000 P c42e0c000a9b4b53a34d41a257df494d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "295ba34a46a74756a017dbe2add34dbb" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 37179 } } peers { permanent_uuid: "198952c9f5644bc5a514d3f6d8a6a871" member_type: VOTER last_known_addr { host: "127.14.103.61" port: 35785 } attrs { promote: false } } peers { permanent_uuid: "c42e0c000a9b4b53a34d41a257df494d" member_type: NON_VOTER last_known_addr { host: "127.14.103.60" port: 40443 } attrs { promote: true } }
I20260812 02:32:18.252321 22455 raft_consensus.cc:729] T 00000000000000000000000000000000 P c42e0c000a9b4b53a34d41a257df494d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c42e0c000a9b4b53a34d41a257df494d, State: Initialized, Role: LEARNER
I20260812 02:32:18.252517 22455 consensus_queue.cc:260] T 00000000000000000000000000000000 P c42e0c000a9b4b53a34d41a257df494d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "295ba34a46a74756a017dbe2add34dbb" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 37179 } } peers { permanent_uuid: "198952c9f5644bc5a514d3f6d8a6a871" member_type: VOTER last_known_addr { host: "127.14.103.61" port: 35785 } attrs { promote: false } } peers { permanent_uuid: "c42e0c000a9b4b53a34d41a257df494d" member_type: NON_VOTER last_known_addr { host: "127.14.103.60" port: 40443 } attrs { promote: true } }
I20260812 02:32:18.253015 22458 sys_catalog.cc:455] T 00000000000000000000000000000000 P c42e0c000a9b4b53a34d41a257df494d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "295ba34a46a74756a017dbe2add34dbb" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 37179 } } peers { permanent_uuid: "198952c9f5644bc5a514d3f6d8a6a871" member_type: VOTER last_known_addr { host: "127.14.103.61" port: 35785 } attrs { promote: false } } peers { permanent_uuid: "c42e0c000a9b4b53a34d41a257df494d" member_type: NON_VOTER last_known_addr { host: "127.14.103.60" port: 40443 } attrs { promote: true } } }
I20260812 02:32:18.253158 22455 sys_catalog.cc:564] T 00000000000000000000000000000000 P c42e0c000a9b4b53a34d41a257df494d [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:32:18.253232 22458 sys_catalog.cc:458] T 00000000000000000000000000000000 P c42e0c000a9b4b53a34d41a257df494d [sys.catalog]: This master's current role is: LEARNER
I20260812 02:32:18.256748 22469 catalog_manager.cc:1259] Loaded cluster ID: 6e4d29f3e17c4d179f9de01bd0f0be77
I20260812 02:32:18.256834 22469 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c42e0c000a9b4b53a34d41a257df494d: loading cluster ID for follower catalog manager: success
I20260812 02:32:18.259992 22469 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c42e0c000a9b4b53a34d41a257df494d: acquiring CA information for follower catalog manager: success
I20260812 02:32:18.262460 22469 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c42e0c000a9b4b53a34d41a257df494d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260812 02:32:18.264446 22368 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:32:18.264730 22368 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:32:18.264778 22368 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260812 02:32:18.266206 22038 heartbeater.cc:503] Master 127.14.103.62:37179 was elected leader, sending a full tablet report...
W20260812 02:32:18.270659 22368 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:32:18.270747 22368 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:32:18.270844 22368 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.103.1
I20260812 02:32:18.277473 22368 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.103.20:46127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/krb5kdc/oryx_127.14.103.1.keytab
--principal=oryx/127.14.103.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.103.1:46041
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.103.1
--webserver_port=41597
--tserver_master_addrs=127.14.103.62:37179,127.14.103.61:35785,127.14.103.60:40443
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22368
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.103.1
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:32:18.278760 22368 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:32:18.280009 22368 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260812 02:32:18.295542 22368 server_base.cc:1034] running on GCE node
W20260812 02:32:18.295488 22478 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:32:18.295396 22477 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:32:18.295404 22480 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:32:18.296247 22368 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:32:18.297000 22368 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:32:18.298230 22368 hybrid_clock.cc:648] HybridClock initialized: now 1786501938298215 us; error 44 us; skew 500 ppm
Aug 12 02:32:18 dist-test-slave-2w3w krb5kdc[21470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501938, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.103.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:32:18.302898 22368 init.cc:377] Logged in from keytab as oryx/127.14.103.1@KRBTEST.COM (short username oryx)
I20260812 02:32:18.305008 22368 webserver.cc:460] Webserver started at http://127.14.103.1:41597/ using document root <none> and password file <none>
I20260812 02:32:18.306244 22368 fs_manager.cc:359] Metadata directory not provided
I20260812 02:32:18.306541 22368 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:32:18.312112 22368 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260812 02:32:18.314860 22489 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:32:18.316632 22368 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260812 02:32:18.316807 22368 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/minicluster-data/ts-0/wal
uuid: "88fa3b5bb4714fa691fcc87e8d327b8f"
format_stamp: "Formatted at 2026-08-12 02:32:16 on dist-test-slave-2w3w"
server_key: "6dbacc41c451a7d910bc00ecae9222d4"
server_key_iv: "bb556621c4acf5ab9cab4bce5da95769"
server_key_version: "encryptionkey@0"
I20260812 02:32:18.317478 22368 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:32:18.335655 22368 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:32:18.341004 22368 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:32:18.341306 22368 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:32:18.342231 22368 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:32:18.343847 22368 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:32:18.343916 22368 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:32:18.343961 22368 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:32:18.343976 22368 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:32:18.356745 22368 rpc_server.cc:307] RPC server started. Bound to: 127.14.103.1:46041
I20260812 02:32:18.356772 22621 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.103.1:46041 every 8 connection(s)
I20260812 02:32:18.357890 22368 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/minicluster-data/ts-0/data/info.pb
I20260812 02:32:18.361974 14748 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbep4M4/build/debug/bin/kudu as pid 22368
I20260812 02:32:18.362498 14748 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbep4M4/build/debug/bin/kudu with pid 21744
Aug 12 02:32:18 dist-test-slave-2w3w krb5kdc[21470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501938, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.103.1@KRBTEST.COM for oryx/127.14.103.61@KRBTEST.COM
Aug 12 02:32:18 dist-test-slave-2w3w krb5kdc[21470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501938, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.103.1@KRBTEST.COM for oryx/127.14.103.62@KRBTEST.COM
I20260812 02:32:18.383728 22343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:18.361580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.1:55539 (local address 127.14.103.61:35785)
0812 02:32:18.361745 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:18.361750 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:18.367892 (+ 6142us) server_negotiation.cc:407] Connection header received
0812 02:32:18.368050 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:18.368053 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:18.368115 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:18.368203 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:32:18.369390 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:18.370434 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:18.372908 (+ 2474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:18.373275 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:18.373369 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:18.377188 (+ 3819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:32:18.377193 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:32:18.377195 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:32:18.377229 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:32:18.379850 (+ 2621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:18.380663 (+ 813us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:18.380666 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:18.380668 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:18.380753 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:18.381157 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:18.381161 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:18.381162 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:18.381379 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:32:18.381504 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0812 02:32:18.383072 (+ 1568us) server_negotiation.cc:299] Negotiation successful
0812 02:32:18.383537 (+ 465us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260812 02:32:18.384477 21567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:18.360172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.1:49549 (local address 127.14.103.62:37179)
0812 02:32:18.360475 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:18.360481 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:18.364140 (+ 3659us) server_negotiation.cc:407] Connection header received
0812 02:32:18.366736 (+ 2596us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:18.366741 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:18.366828 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:18.367395 (+ 567us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:32:18.369467 (+ 2072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:18.370451 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:18.373056 (+ 2605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:18.373554 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:18.373672 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:18.378532 (+ 4860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:32:18.378539 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:32:18.378543 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:32:18.378585 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:32:18.380235 (+ 1650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:18.381513 (+ 1278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:18.381520 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:18.381524 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:18.381623 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:18.382366 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:18.382369 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:18.382371 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:18.382720 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:32:18.382890 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0812 02:32:18.383426 (+ 536us) server_negotiation.cc:299] Negotiation successful
0812 02:32:18.383614 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260812 02:32:18.386101 22629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:18.361009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.61:35785 (local address 127.14.103.1:55539)
0812 02:32:18.367696 (+ 6687us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:18.367737 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0812 02:32:18.367932 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:32:18.368219 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:32:18.368222 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:32:18.368462 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:32:18.369203 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:18.369222 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:18.370800 (+ 1578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:18.370803 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:18.372682 (+ 1879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:18.372705 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:18.373640 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:18.373647 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:18.373902 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:18.374718 (+ 816us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:32:18.374762 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:32:18.376916 (+ 2154us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:32:18.380050 (+ 3134us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:32:18.380061 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:32:18.380087 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:32:18.380526 (+ 439us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:32:18.380858 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:32:18.380862 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:32:18.380864 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:32:18.381044 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:32:18.381992 (+ 948us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:32:18.381998 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:32:18.382700 (+ 702us) client_negotiation.cc:769] Sending connection context
0812 02:32:18.382998 (+ 298us) client_negotiation.cc:240] Negotiation successful
0812 02:32:18.383706 (+ 708us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6452,"mutex_wait_us":326,"spinlock_wait_cycles":2432,"thread_start_us":246,"threads_started":1}
I20260812 02:32:18.386197 22628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:18.361008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:37179 (local address 127.14.103.1:49549)
0812 02:32:18.363588 (+ 2580us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:18.363732 (+ 144us) client_negotiation.cc:174] Beginning negotiation
0812 02:32:18.366231 (+ 2499us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:32:18.367461 (+ 1230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:32:18.367476 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:32:18.368478 (+ 1002us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:32:18.369208 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:18.369226 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:18.370757 (+ 1531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:18.370764 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:18.372824 (+ 2060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:18.372843 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:18.373918 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:18.373925 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:18.374079 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:18.375530 (+ 1451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:32:18.375553 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:32:18.378180 (+ 2627us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:32:18.380466 (+ 2286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:32:18.380474 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:32:18.380480 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:32:18.381341 (+ 861us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:32:18.381940 (+ 599us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:32:18.381947 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:32:18.381952 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:32:18.382097 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:32:18.382937 (+ 840us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:32:18.382943 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:32:18.383157 (+ 214us) client_negotiation.cc:769] Sending connection context
0812 02:32:18.385437 (+ 2280us) client_negotiation.cc:240] Negotiation successful
0812 02:32:18.385684 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1502,"mutex_wait_us":83,"spinlock_wait_cycles":2816,"thread_start_us":355,"threads_started":1}
I20260812 02:32:18.388537 22625 heartbeater.cc:347] Connected to a master server at 127.14.103.61:35785
I20260812 02:32:18.389019 22625 heartbeater.cc:464] Registering TS with master...
I20260812 02:32:18.389815 22625 heartbeater.cc:511] Master 127.14.103.61:35785 requested a full tablet report, sending...
I20260812 02:32:18.392300 22194 ts_manager.cc:194] Registered new tserver with Master: 88fa3b5bb4714fa691fcc87e8d327b8f (127.14.103.1:46041)
I20260812 02:32:18.393517 22623 heartbeater.cc:347] Connected to a master server at 127.14.103.62:37179
I20260812 02:32:18.393627 22623 heartbeater.cc:464] Registering TS with master...
I20260812 02:32:18.393904 22623 heartbeater.cc:511] Master 127.14.103.62:37179 requested a full tablet report, sending...
I20260812 02:32:18.395715 21520 ts_manager.cc:194] Re-registered known tserver with Master: 88fa3b5bb4714fa691fcc87e8d327b8f (127.14.103.1:46041)
Aug 12 02:32:18 dist-test-slave-2w3w krb5kdc[21470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501938, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.103.1@KRBTEST.COM for oryx/127.14.103.60@KRBTEST.COM
I20260812 02:32:18.396677 21520 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.14.103.1@KRBTEST.COM'} at 127.14.103.1:49549
I20260812 02:32:18.404663 22627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:18.365832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.60:40443 (local address 127.14.103.1:59867)
0812 02:32:18.376134 (+ 10302us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:18.376151 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 02:32:18.376279 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:32:18.390181 (+ 13902us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:32:18.390185 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:32:18.390429 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:32:18.390585 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:18.390593 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:18.391867 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:18.391874 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:18.392925 (+ 1051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:18.392944 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:18.393857 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:18.393863 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:18.394091 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:18.394798 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:32:18.394815 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:32:18.397010 (+ 2195us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:32:18.399926 (+ 2916us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:32:18.399929 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:32:18.399932 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:32:18.400256 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:32:18.403431 (+ 3175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:32:18.403435 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:32:18.403437 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:32:18.403557 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:32:18.404054 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:32:18.404057 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:32:18.404159 (+ 102us) client_negotiation.cc:769] Sending connection context
0812 02:32:18.404248 (+ 89us) client_negotiation.cc:240] Negotiation successful
0812 02:32:18.404451 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10099,"thread_start_us":555,"threads_started":1}
I20260812 02:32:18.405740 22630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:18.389206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.1:59867 (local address 127.14.103.60:40443)
0812 02:32:18.389700 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:18.389723 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:18.389742 (+ 19us) server_negotiation.cc:407] Connection header received
0812 02:32:18.389809 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:18.389823 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:18.389877 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:18.390715 (+ 838us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:32:18.390824 (+ 109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:18.391693 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:18.393121 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:18.393515 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:18.394414 (+ 899us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:18.397263 (+ 2849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:32:18.397269 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:32:18.397275 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:32:18.397313 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:32:18.399718 (+ 2405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:18.403006 (+ 3288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:18.403012 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:18.403014 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:18.403275 (+ 261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:18.403678 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:18.403681 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:18.403683 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:18.403921 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:32:18.405292 (+ 1371us) server_negotiation.cc:1035] Waiting for connection context
0812 02:32:18.405396 (+ 104us) server_negotiation.cc:299] Negotiation successful
0812 02:32:18.405563 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":124,"threads_started":1}
I20260812 02:32:18.406589 22622 heartbeater.cc:347] Connected to a master server at 127.14.103.60:40443
I20260812 02:32:18.406747 22622 heartbeater.cc:464] Registering TS with master...
I20260812 02:32:18.407104 22622 heartbeater.cc:511] Master 127.14.103.60:40443 requested a full tablet report, sending...
I20260812 02:32:18.408819 22406 ts_manager.cc:194] Registered new tserver with Master: 88fa3b5bb4714fa691fcc87e8d327b8f (127.14.103.1:46041)
I20260812 02:32:18.411402 14748 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbep4M4/build/debug/bin/kudu
/tmp/dist-test-taskbep4M4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.103.2:35055
--local_ip_for_outbound_sockets=127.14.103.2
--tserver_master_addrs=127.14.103.62:37179
--webserver_port=35535
--webserver_interface=127.14.103.2
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/krb5kdc/oryx_127.14.103.2.keytab
--principal=oryx/127.14.103.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.103.20:46127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.14.103.62:37179,127.14.103.61:35785
--tserver_master_addrs=127.14.103.62:37179,127.14.103.61:35785,127.14.103.60:40443 with env {KRB5CCNAME=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 02:32:18.493023 22152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:18.488659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.60:40443 (local address 127.0.0.1:48900)
0812 02:32:18.488831 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:18.488850 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 02:32:18.488983 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:32:18.489262 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:32:18.489265 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:32:18.489283 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:32:18.489394 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:18.489401 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:18.490470 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:18.490474 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:18.491702 (+ 1228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:18.491717 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:18.492530 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:18.492534 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:18.492630 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:18.492659 (+ 29us) client_negotiation.cc:769] Sending connection context
0812 02:32:18.492747 (+ 88us) client_negotiation.cc:240] Negotiation successful
0812 02:32:18.492847 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68}
I20260812 02:32:18.493350 22630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:18.488728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48900 (local address 127.14.103.60:40443)
0812 02:32:18.488869 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:18.488873 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:18.488993 (+ 120us) server_negotiation.cc:407] Connection header received
0812 02:32:18.489067 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:18.489069 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:18.489136 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:18.489227 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:32:18.489564 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:18.490339 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:18.491872 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:18.492375 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:18.492485 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:18.492609 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0812 02:32:18.492899 (+ 290us) server_negotiation.cc:299] Negotiation successful
0812 02:32:18.493058 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260812 02:32:18.494395 22425 raft_consensus.cc:1203] T 00000000000000000000000000000000 P c42e0c000a9b4b53a34d41a257df494d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260812 02:32:18.495347 22458 sys_catalog.cc:455] T 00000000000000000000000000000000 P c42e0c000a9b4b53a34d41a257df494d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 295ba34a46a74756a017dbe2add34dbb. Latest consensus state: current_term: 1 leader_uuid: "295ba34a46a74756a017dbe2add34dbb" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "295ba34a46a74756a017dbe2add34dbb" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 37179 } } peers { permanent_uuid: "198952c9f5644bc5a514d3f6d8a6a871" member_type: VOTER last_known_addr { host: "127.14.103.61" port: 35785 } attrs { promote: false } } peers { permanent_uuid: "c42e0c000a9b4b53a34d41a257df494d" member_type: NON_VOTER last_known_addr { host: "127.14.103.60" port: 40443 } attrs { promote: true } } }
I20260812 02:32:18.495556 22458 sys_catalog.cc:458] T 00000000000000000000000000000000 P c42e0c000a9b4b53a34d41a257df494d [sys.catalog]: This master's current role is: LEARNER
W20260812 02:32:18.571594 22631 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:32:18.571985 22631 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:32:18.572088 22631 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:32:18.577193 22631 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:32:18.577318 22631 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:32:18.577426 22631 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.103.2
I20260812 02:32:18.584079 22631 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.103.20:46127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/krb5kdc/oryx_127.14.103.2.keytab
--principal=oryx/127.14.103.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.103.2:35055
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.103.2
--webserver_port=35535
--tserver_master_addrs=127.14.103.62:37179,127.14.103.61:35785,127.14.103.60:40443
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22631
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.103.2
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:32:18.585383 22631 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:32:18.586530 22631 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:32:18.597956 22640 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:32:18.597977 22639 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:32:18.598078 22643 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:32:18.598271 22631 server_base.cc:1034] running on GCE node
I20260812 02:32:18.598892 22631 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:32:18.599768 22631 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:32:18.601006 22631 hybrid_clock.cc:648] HybridClock initialized: now 1786501938600987 us; error 36 us; skew 500 ppm
Aug 12 02:32:18 dist-test-slave-2w3w krb5kdc[21470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501938, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.103.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:32:18.604594 22631 init.cc:377] Logged in from keytab as oryx/127.14.103.2@KRBTEST.COM (short username oryx)
I20260812 02:32:18.605638 22631 webserver.cc:460] Webserver started at http://127.14.103.2:35535/ using document root <none> and password file <none>
I20260812 02:32:18.606271 22631 fs_manager.cc:359] Metadata directory not provided
I20260812 02:32:18.606401 22631 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:32:18.611553 22631 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260812 02:32:18.613996 22651 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:32:18.615556 22631 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 02:32:18.615725 22631 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/minicluster-data/ts-1/wal
uuid: "0294bdfc2c3e476fb7c39c969dee950b"
format_stamp: "Formatted at 2026-08-12 02:32:16 on dist-test-slave-2w3w"
server_key: "21a3122387229cbbdf72d316e8dec9b4"
server_key_iv: "57a3b60bc912038b69a3aa6326528f92"
server_key_version: "encryptionkey@0"
I20260812 02:32:18.616410 22631 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:32:18.636251 22631 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:32:18.640748 22631 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:32:18.641058 22631 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:32:18.641767 22631 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:32:18.642805 22631 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:32:18.642884 22631 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:32:18.642988 22631 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:32:18.643046 22631 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260812 02:32:18.654829 22631 rpc_server.cc:307] RPC server started. Bound to: 127.14.103.2:35055
I20260812 02:32:18.654886 22781 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.103.2:35055 every 8 connection(s)
I20260812 02:32:18.656073 22631 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/minicluster-data/ts-1/data/info.pb
I20260812 02:32:18.664711 14748 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbep4M4/build/debug/bin/kudu as pid 22631
I20260812 02:32:18.665001 14748 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbep4M4/build/debug/bin/kudu with pid 21891
Aug 12 02:32:18 dist-test-slave-2w3w krb5kdc[21470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501938, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.103.2@KRBTEST.COM for oryx/127.14.103.61@KRBTEST.COM
Aug 12 02:32:18 dist-test-slave-2w3w krb5kdc[21470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501938, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.103.2@KRBTEST.COM for oryx/127.14.103.60@KRBTEST.COM
Aug 12 02:32:18 dist-test-slave-2w3w krb5kdc[21470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501938, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.103.2@KRBTEST.COM for oryx/127.14.103.62@KRBTEST.COM
I20260812 02:32:18.680322 22343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:18.658238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.2:57689 (local address 127.14.103.61:35785)
0812 02:32:18.658377 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:18.658382 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:18.660278 (+ 1896us) server_negotiation.cc:407] Connection header received
0812 02:32:18.661611 (+ 1333us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:18.661615 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:18.661669 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:18.661755 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:32:18.663717 (+ 1962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:18.664755 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:18.668743 (+ 3988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:18.669230 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:18.669324 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:18.674940 (+ 5616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:32:18.674947 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:32:18.674953 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:32:18.675018 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:32:18.676900 (+ 1882us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:18.677657 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:18.677661 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:18.677663 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:18.677755 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:18.678230 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:18.678234 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:18.678236 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:18.678517 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:32:18.678601 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0812 02:32:18.679789 (+ 1188us) server_negotiation.cc:299] Negotiation successful
0812 02:32:18.679941 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260812 02:32:18.680322 22630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:18.658269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.2:42351 (local address 127.14.103.60:40443)
0812 02:32:18.658429 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:18.658435 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:18.659785 (+ 1350us) server_negotiation.cc:407] Connection header received
0812 02:32:18.661303 (+ 1518us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:18.661306 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:18.661362 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:18.661453 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:32:18.665193 (+ 3740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:18.665972 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:18.670058 (+ 4086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:18.670454 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:18.670596 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:18.674644 (+ 4048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:32:18.674650 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:32:18.674655 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:32:18.674692 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:32:18.676741 (+ 2049us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:18.677658 (+ 917us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:18.677661 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:18.677663 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:18.677755 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:18.678230 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:18.678234 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:18.678236 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:18.678507 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:32:18.678601 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0812 02:32:18.679789 (+ 1188us) server_negotiation.cc:299] Negotiation successful
0812 02:32:18.679951 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260812 02:32:18.682704 21567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:18.660165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.2:49001 (local address 127.14.103.62:37179)
0812 02:32:18.660299 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:18.660303 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:18.660677 (+ 374us) server_negotiation.cc:407] Connection header received
0812 02:32:18.661298 (+ 621us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:18.661302 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:18.661362 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:18.661423 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:32:18.663862 (+ 2439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:18.664968 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:18.668761 (+ 3793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:18.669295 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:18.669433 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:18.675678 (+ 6245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:32:18.675687 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:32:18.675693 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:32:18.676227 (+ 534us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:32:18.678841 (+ 2614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:18.679581 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:18.679588 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:18.679594 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:18.679710 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:18.680166 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:18.680173 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:18.680177 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:18.680494 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:32:18.680613 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0812 02:32:18.681891 (+ 1278us) server_negotiation.cc:299] Negotiation successful
0812 02:32:18.682267 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260812 02:32:18.684479 22789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:18.658743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.60:40443 (local address 127.14.103.2:42351)
0812 02:32:18.659573 (+ 830us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:18.659636 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0812 02:32:18.660980 (+ 1344us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:32:18.661684 (+ 704us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:32:18.661692 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:32:18.662435 (+ 743us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:32:18.663462 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:18.663481 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:18.667816 (+ 4335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:18.667832 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:18.668935 (+ 1103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:18.668947 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:18.670625 (+ 1678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:18.670630 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:18.670740 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:18.671224 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:32:18.671240 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:32:18.674149 (+ 2909us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:32:18.676933 (+ 2784us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:32:18.676944 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:32:18.676969 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:32:18.677525 (+ 556us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:32:18.677864 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:32:18.677866 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:32:18.677868 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:32:18.678108 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:32:18.678663 (+ 555us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:32:18.678671 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:32:18.679445 (+ 774us) client_negotiation.cc:769] Sending connection context
0812 02:32:18.680587 (+ 1142us) client_negotiation.cc:240] Negotiation successful
0812 02:32:18.682066 (+ 1479us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":568,"mutex_wait_us":1070,"spinlock_wait_cycles":29952,"thread_start_us":391,"threads_started":1}
I20260812 02:32:18.685139 22790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:18.658743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.61:35785 (local address 127.14.103.2:57689)
0812 02:32:18.660050 (+ 1307us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:18.660067 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 02:32:18.660980 (+ 913us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:32:18.661796 (+ 816us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:32:18.661798 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:32:18.662449 (+ 651us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:32:18.663454 (+ 1005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:18.663481 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:18.665284 (+ 1803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:18.665291 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:18.668465 (+ 3174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:18.668491 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:18.669604 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:18.669611 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:18.669897 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:18.671110 (+ 1213us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:32:18.671130 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:32:18.674710 (+ 3580us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:32:18.677021 (+ 2311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:32:18.677027 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:32:18.677032 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:32:18.677525 (+ 493us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:32:18.677864 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:32:18.677866 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:32:18.677868 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:32:18.678108 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:32:18.678663 (+ 555us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:32:18.678671 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:32:18.679461 (+ 790us) client_negotiation.cc:769] Sending connection context
0812 02:32:18.680573 (+ 1112us) client_negotiation.cc:240] Negotiation successful
0812 02:32:18.682017 (+ 1444us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1179,"mutex_wait_us":354,"spinlock_wait_cycles":27264,"thread_start_us":347,"threads_started":1}
I20260812 02:32:18.686542 22782 heartbeater.cc:347] Connected to a master server at 127.14.103.60:40443
I20260812 02:32:18.684536 22791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:18.660058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:37179 (local address 127.14.103.2:49001)
0812 02:32:18.660533 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:18.660547 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 02:32:18.661013 (+ 466us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:32:18.661684 (+ 671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:32:18.661694 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:32:18.662435 (+ 741us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:32:18.663542 (+ 1107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:18.663571 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:18.665310 (+ 1739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:18.665319 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:18.668465 (+ 3146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:18.668493 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:18.669681 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:18.669689 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:18.669908 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:18.671275 (+ 1367us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:32:18.671299 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:32:18.674783 (+ 3484us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:32:18.679063 (+ 4280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:32:18.679069 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:32:18.679074 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:32:18.679423 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:32:18.679871 (+ 448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:32:18.679877 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:32:18.679882 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:32:18.680002 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:32:18.681262 (+ 1260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:32:18.681268 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:32:18.681584 (+ 316us) client_negotiation.cc:769] Sending connection context
0812 02:32:18.683611 (+ 2027us) client_negotiation.cc:240] Negotiation successful
0812 02:32:18.683936 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":397,"mutex_wait_us":308,"spinlock_wait_cycles":26496,"thread_start_us":302,"threads_started":1}
I20260812 02:32:18.687068 22782 heartbeater.cc:464] Registering TS with master...
I20260812 02:32:18.689276 22782 heartbeater.cc:511] Master 127.14.103.60:40443 requested a full tablet report, sending...
I20260812 02:32:18.690914 22783 heartbeater.cc:347] Connected to a master server at 127.14.103.62:37179
I20260812 02:32:18.691267 22783 heartbeater.cc:464] Registering TS with master...
I20260812 02:32:18.691570 22783 heartbeater.cc:511] Master 127.14.103.62:37179 requested a full tablet report, sending...
I20260812 02:32:18.691529 22406 ts_manager.cc:194] Registered new tserver with Master: 0294bdfc2c3e476fb7c39c969dee950b (127.14.103.2:35055)
I20260812 02:32:18.692503 21520 ts_manager.cc:194] Re-registered known tserver with Master: 0294bdfc2c3e476fb7c39c969dee950b (127.14.103.2:35055)
I20260812 02:32:18.693624 21520 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.14.103.2@KRBTEST.COM'} at 127.14.103.2:49001
I20260812 02:32:18.699793 22786 heartbeater.cc:347] Connected to a master server at 127.14.103.61:35785
I20260812 02:32:18.700033 22786 heartbeater.cc:464] Registering TS with master...
I20260812 02:32:18.700322 22786 heartbeater.cc:511] Master 127.14.103.61:35785 requested a full tablet report, sending...
I20260812 02:32:18.701359 22194 ts_manager.cc:194] Registered new tserver with Master: 0294bdfc2c3e476fb7c39c969dee950b (127.14.103.2:35055)
I20260812 02:32:18.702608 14748 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbep4M4/build/debug/bin/kudu
/tmp/dist-test-taskbep4M4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.103.3:36861
--local_ip_for_outbound_sockets=127.14.103.3
--tserver_master_addrs=127.14.103.62:37179
--webserver_port=32941
--webserver_interface=127.14.103.3
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/krb5kdc/oryx_127.14.103.3.keytab
--principal=oryx/127.14.103.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.103.20:46127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.14.103.62:37179,127.14.103.61:35785
--tserver_master_addrs=127.14.103.62:37179,127.14.103.61:35785,127.14.103.60:40443 with env {KRB5CCNAME=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:32:18.849426 22792 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:32:18.849867 22792 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:32:18.849939 22792 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:32:18.854844 22792 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:32:18.854923 22792 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:32:18.855057 22792 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.103.3
I20260812 02:32:18.861004 22792 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.103.20:46127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/krb5kdc/oryx_127.14.103.3.keytab
--principal=oryx/127.14.103.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.103.3:36861
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.103.3
--webserver_port=32941
--tserver_master_addrs=127.14.103.62:37179,127.14.103.61:35785,127.14.103.60:40443
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22792
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.103.3
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:32:18.862181 22792 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:32:18.863389 22792 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:32:18.876586 22801 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:32:18.876762 22798 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:32:18.876789 22797 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:32:18.877039 22792 server_base.cc:1034] running on GCE node
I20260812 02:32:18.877597 22792 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:32:18.878348 22792 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:32:18.879568 22792 hybrid_clock.cc:648] HybridClock initialized: now 1786501938879548 us; error 40 us; skew 500 ppm
Aug 12 02:32:18 dist-test-slave-2w3w krb5kdc[21470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501938, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.103.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:32:18.883226 22792 init.cc:377] Logged in from keytab as oryx/127.14.103.3@KRBTEST.COM (short username oryx)
I20260812 02:32:18.884276 22792 webserver.cc:460] Webserver started at http://127.14.103.3:32941/ using document root <none> and password file <none>
I20260812 02:32:18.884912 22792 fs_manager.cc:359] Metadata directory not provided
I20260812 02:32:18.885036 22792 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:32:18.889652 22792 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260812 02:32:18.892390 22809 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:32:18.894346 22792 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 02:32:18.894522 22792 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/minicluster-data/ts-2/wal
uuid: "9abf9de06df64b46b5db9181d885a31a"
format_stamp: "Formatted at 2026-08-12 02:32:17 on dist-test-slave-2w3w"
server_key: "974bacd2188c10a38b65af0c95095011"
server_key_iv: "f299871e4bf5c14d75f102346ee98d22"
server_key_version: "encryptionkey@0"
I20260812 02:32:18.895262 22792 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:32:18.939841 22792 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:32:18.945766 22792 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:32:18.946327 22792 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:32:18.947479 22792 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:32:18.948740 22792 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:32:18.948840 22792 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:32:18.948935 22792 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:32:18.948982 22792 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:32:18.960925 22792 rpc_server.cc:307] RPC server started. Bound to: 127.14.103.3:36861
I20260812 02:32:18.960932 22942 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.103.3:36861 every 8 connection(s)
I20260812 02:32:18.962572 22792 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/minicluster-data/ts-2/data/info.pb
I20260812 02:32:18.965003 14748 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbep4M4/build/debug/bin/kudu as pid 22792
Aug 12 02:32:18 dist-test-slave-2w3w krb5kdc[21470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501938, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.103.3@KRBTEST.COM for oryx/127.14.103.62@KRBTEST.COM
I20260812 02:32:18.978188 22156 sys_catalog.cc:455] T 00000000000000000000000000000000 P 295ba34a46a74756a017dbe2add34dbb [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "295ba34a46a74756a017dbe2add34dbb" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "295ba34a46a74756a017dbe2add34dbb" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 37179 } } peers { permanent_uuid: "198952c9f5644bc5a514d3f6d8a6a871" member_type: VOTER last_known_addr { host: "127.14.103.61" port: 35785 } attrs { promote: false } } peers { permanent_uuid: "c42e0c000a9b4b53a34d41a257df494d" member_type: NON_VOTER last_known_addr { host: "127.14.103.60" port: 40443 } attrs { promote: true } } }
I20260812 02:32:18.978367 22156 sys_catalog.cc:458] T 00000000000000000000000000000000 P 295ba34a46a74756a017dbe2add34dbb [sys.catalog]: This master's current role is: LEADER
I20260812 02:32:18.979758 22156 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 295ba34a46a74756a017dbe2add34dbb: attempting to promote NON_VOTER c42e0c000a9b4b53a34d41a257df494d to VOTER
Aug 12 02:32:18 dist-test-slave-2w3w krb5kdc[21470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501938, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.103.3@KRBTEST.COM for oryx/127.14.103.60@KRBTEST.COM
I20260812 02:32:18.980237 22156 consensus_queue.cc:237] T 00000000000000000000000000000000 P 295ba34a46a74756a017dbe2add34dbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "295ba34a46a74756a017dbe2add34dbb" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 37179 } } peers { permanent_uuid: "198952c9f5644bc5a514d3f6d8a6a871" member_type: VOTER last_known_addr { host: "127.14.103.61" port: 35785 } attrs { promote: false } } peers { permanent_uuid: "c42e0c000a9b4b53a34d41a257df494d" member_type: VOTER last_known_addr { host: "127.14.103.60" port: 40443 } attrs { promote: false } }
I20260812 02:32:18.982581 22425 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c42e0c000a9b4b53a34d41a257df494d [term 1 LEARNER]: Refusing update from remote peer 295ba34a46a74756a017dbe2add34dbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
Aug 12 02:32:18 dist-test-slave-2w3w krb5kdc[21470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501938, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.103.3@KRBTEST.COM for oryx/127.14.103.61@KRBTEST.COM
I20260812 02:32:18.983148 22213 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 198952c9f5644bc5a514d3f6d8a6a871 [term 1 FOLLOWER]: Refusing update from remote peer 295ba34a46a74756a017dbe2add34dbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260812 02:32:18.983131 22156 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 295ba34a46a74756a017dbe2add34dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "c42e0c000a9b4b53a34d41a257df494d" member_type: VOTER last_known_addr { host: "127.14.103.60" port: 40443 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260812 02:32:18.983640 22156 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 295ba34a46a74756a017dbe2add34dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "198952c9f5644bc5a514d3f6d8a6a871" member_type: VOTER last_known_addr { host: "127.14.103.61" port: 35785 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260812 02:32:18.985682 22961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:18.968859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47244 (local address 127.14.103.62:37179)
0812 02:32:18.970640 (+ 1781us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:18.970645 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:18.970662 (+ 17us) server_negotiation.cc:407] Connection header received
0812 02:32:18.970709 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:18.970713 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:18.970760 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:18.971881 (+ 1121us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:32:18.971977 (+ 96us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:18.972714 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:18.974110 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:18.974510 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:18.975639 (+ 1129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:18.976943 (+ 1304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:32:18.976947 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:32:18.976950 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:32:18.976981 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:32:18.979266 (+ 2285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:18.980304 (+ 1038us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:18.980308 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:18.980310 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:18.980406 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:18.981825 (+ 1419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:18.981829 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:18.981831 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:18.982670 (+ 839us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:32:18.985131 (+ 2461us) server_negotiation.cc:1035] Waiting for connection context
0812 02:32:18.985224 (+ 93us) server_negotiation.cc:299] Negotiation successful
0812 02:32:18.985397 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":1695,"thread_start_us":81,"threads_started":1}
I20260812 02:32:18.986265 22630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:18.967715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.3:56261 (local address 127.14.103.60:40443)
0812 02:32:18.967850 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:18.967855 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:18.968055 (+ 200us) server_negotiation.cc:407] Connection header received
0812 02:32:18.968587 (+ 532us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:18.968595 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:18.968796 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:18.968881 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:32:18.971249 (+ 2368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:18.972219 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:18.974145 (+ 1926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:18.974530 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:18.974621 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:18.981033 (+ 6412us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:32:18.981041 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:32:18.981047 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:32:18.981084 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:32:18.982798 (+ 1714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:18.984002 (+ 1204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:18.984006 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:18.984007 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:18.984106 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:18.984567 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:18.984571 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:18.984572 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:18.984795 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:32:18.984882 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0812 02:32:18.985928 (+ 1046us) server_negotiation.cc:299] Negotiation successful
0812 02:32:18.986067 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":48}
I20260812 02:32:18.988277 22955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:18.967344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.60:40443 (local address 127.14.103.3:56261)
0812 02:32:18.967946 (+ 602us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:18.967959 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 02:32:18.968292 (+ 333us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:32:18.969276 (+ 984us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:32:18.969285 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:32:18.970007 (+ 722us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:32:18.971038 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:18.971051 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:18.972369 (+ 1318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:18.972372 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:18.974002 (+ 1630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:18.974017 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:18.976262 (+ 2245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:18.976266 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:18.976359 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:18.977391 (+ 1032us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:32:18.977409 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:32:18.980772 (+ 3363us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:32:18.983401 (+ 2629us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:32:18.983409 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:32:18.983432 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:32:18.983879 (+ 447us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:32:18.984231 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:32:18.984235 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:32:18.984237 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:32:18.984437 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:32:18.984955 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:32:18.984962 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:32:18.985348 (+ 386us) client_negotiation.cc:769] Sending connection context
0812 02:32:18.986519 (+ 1171us) client_negotiation.cc:240] Negotiation successful
0812 02:32:18.987028 (+ 509us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"mutex_wait_us":215,"thread_start_us":84,"threads_started":1}
I20260812 02:32:18.988430 22425 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c42e0c000a9b4b53a34d41a257df494d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, c42e0c000a9b4b53a34d41a257df494d (127.14.103.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "295ba34a46a74756a017dbe2add34dbb" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 37179 } } peers { permanent_uuid: "198952c9f5644bc5a514d3f6d8a6a871" member_type: VOTER last_known_addr { host: "127.14.103.61" port: 35785 } attrs { promote: false } } peers { permanent_uuid: "c42e0c000a9b4b53a34d41a257df494d" member_type: VOTER last_known_addr { host: "127.14.103.60" port: 40443 } attrs { promote: false } } }
I20260812 02:32:18.989898 22959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:18.968434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35120 (local address 127.14.103.61:35785)
0812 02:32:18.969590 (+ 1156us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:18.969594 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:18.969609 (+ 15us) server_negotiation.cc:407] Connection header received
0812 02:32:18.969655 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:18.969660 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:18.969714 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:18.969825 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:32:18.971435 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:18.972245 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:18.976592 (+ 4347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:18.976949 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:18.977076 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:18.979177 (+ 2101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:32:18.979181 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:32:18.979184 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:32:18.979212 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:32:18.986711 (+ 7499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:18.987427 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:18.987430 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:18.987433 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:18.987533 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:18.988016 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:18.988019 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:18.988021 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:18.988291 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:32:18.989462 (+ 1171us) server_negotiation.cc:1035] Waiting for connection context
0812 02:32:18.989550 (+ 88us) server_negotiation.cc:299] Negotiation successful
0812 02:32:18.989721 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1072,"thread_start_us":290,"threads_started":1}
I20260812 02:32:18.991338 22458 sys_catalog.cc:455] T 00000000000000000000000000000000 P c42e0c000a9b4b53a34d41a257df494d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "295ba34a46a74756a017dbe2add34dbb" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "295ba34a46a74756a017dbe2add34dbb" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 37179 } } peers { permanent_uuid: "198952c9f5644bc5a514d3f6d8a6a871" member_type: VOTER last_known_addr { host: "127.14.103.61" port: 35785 } attrs { promote: false } } peers { permanent_uuid: "c42e0c000a9b4b53a34d41a257df494d" member_type: VOTER last_known_addr { host: "127.14.103.60" port: 40443 } attrs { promote: false } } }
I20260812 02:32:18.991633 22458 sys_catalog.cc:458] T 00000000000000000000000000000000 P c42e0c000a9b4b53a34d41a257df494d [sys.catalog]: This master's current role is: FOLLOWER
I20260812 02:32:18.992357 22943 heartbeater.cc:347] Connected to a master server at 127.14.103.60:40443
I20260812 02:32:18.992655 22943 heartbeater.cc:464] Registering TS with master...
I20260812 02:32:18.993345 22943 heartbeater.cc:511] Master 127.14.103.60:40443 requested a full tablet report, sending...
I20260812 02:32:18.993991 22968 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 295ba34a46a74756a017dbe2add34dbb [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, c42e0c000a9b4b53a34d41a257df494d (127.14.103.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "295ba34a46a74756a017dbe2add34dbb" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 37179 } } peers { permanent_uuid: "198952c9f5644bc5a514d3f6d8a6a871" member_type: VOTER last_known_addr { host: "127.14.103.61" port: 35785 } attrs { promote: false } } peers { permanent_uuid: "c42e0c000a9b4b53a34d41a257df494d" member_type: VOTER last_known_addr { host: "127.14.103.60" port: 40443 } attrs { promote: false } } }
I20260812 02:32:18.994925 22950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:18.965959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:37179 (local address 127.14.103.3:57805)
0812 02:32:18.967402 (+ 1443us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:18.967417 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 02:32:18.968048 (+ 631us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:32:18.969277 (+ 1229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:32:18.969287 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:32:18.969957 (+ 670us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:32:18.971015 (+ 1058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:18.971043 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:18.972222 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:18.972226 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:18.974868 (+ 2642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:18.974878 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:18.975606 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:18.975613 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:18.975848 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:18.976631 (+ 783us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:32:18.976655 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:32:18.985748 (+ 9093us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:32:18.992963 (+ 7215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:32:18.992967 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:32:18.992970 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:32:18.993291 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:32:18.993675 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:32:18.993678 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:32:18.993680 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:32:18.993772 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:32:18.994301 (+ 529us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:32:18.994304 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:32:18.994406 (+ 102us) client_negotiation.cc:769] Sending connection context
0812 02:32:18.994501 (+ 95us) client_negotiation.cc:240] Negotiation successful
0812 02:32:18.994686 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1352,"mutex_wait_us":1989,"spinlock_wait_cycles":13696,"thread_start_us":202,"threads_started":1}
I20260812 02:32:18.995146 22406 ts_manager.cc:194] Registered new tserver with Master: 9abf9de06df64b46b5db9181d885a31a (127.14.103.3:36861)
I20260812 02:32:18.995571 21567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:18.965830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.3:57805 (local address 127.14.103.62:37179)
0812 02:32:18.965953 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:18.965957 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:18.967499 (+ 1542us) server_negotiation.cc:407] Connection header received
0812 02:32:18.968436 (+ 937us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:18.968443 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:18.968510 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:18.968595 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:32:18.971205 (+ 2610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:18.972051 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:18.975051 (+ 3000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:18.975395 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:18.975521 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:18.985916 (+ 10395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:32:18.985921 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:32:18.985924 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:32:18.985949 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:32:18.987799 (+ 1850us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:18.993442 (+ 5643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:18.993446 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:18.993448 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:18.993560 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:18.993876 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:18.993879 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:18.993881 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:18.994146 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:32:18.994255 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0812 02:32:18.995263 (+ 1008us) server_negotiation.cc:299] Negotiation successful
0812 02:32:18.995419 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":51}
I20260812 02:32:18.996276 22966 sys_catalog.cc:455] T 00000000000000000000000000000000 P 295ba34a46a74756a017dbe2add34dbb [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "295ba34a46a74756a017dbe2add34dbb" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "295ba34a46a74756a017dbe2add34dbb" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 37179 } } peers { permanent_uuid: "198952c9f5644bc5a514d3f6d8a6a871" member_type: VOTER last_known_addr { host: "127.14.103.61" port: 35785 } attrs { promote: false } } peers { permanent_uuid: "c42e0c000a9b4b53a34d41a257df494d" member_type: VOTER last_known_addr { host: "127.14.103.60" port: 40443 } attrs { promote: false } } }
I20260812 02:32:18.996397 22966 sys_catalog.cc:458] T 00000000000000000000000000000000 P 295ba34a46a74756a017dbe2add34dbb [sys.catalog]: This master's current role is: LEADER
I20260812 02:32:18.996393 22968 sys_catalog.cc:455] T 00000000000000000000000000000000 P 295ba34a46a74756a017dbe2add34dbb [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "295ba34a46a74756a017dbe2add34dbb" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "295ba34a46a74756a017dbe2add34dbb" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 37179 } } peers { permanent_uuid: "198952c9f5644bc5a514d3f6d8a6a871" member_type: VOTER last_known_addr { host: "127.14.103.61" port: 35785 } attrs { promote: false } } peers { permanent_uuid: "c42e0c000a9b4b53a34d41a257df494d" member_type: VOTER last_known_addr { host: "127.14.103.60" port: 40443 } attrs { promote: false } } }
I20260812 02:32:18.996533 22968 sys_catalog.cc:458] T 00000000000000000000000000000000 P 295ba34a46a74756a017dbe2add34dbb [sys.catalog]: This master's current role is: LEADER
I20260812 02:32:18.996690 22944 heartbeater.cc:347] Connected to a master server at 127.14.103.62:37179
I20260812 02:32:18.996707 22970 sys_catalog.cc:455] T 00000000000000000000000000000000 P 295ba34a46a74756a017dbe2add34dbb [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "295ba34a46a74756a017dbe2add34dbb" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "295ba34a46a74756a017dbe2add34dbb" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 37179 } } peers { permanent_uuid: "198952c9f5644bc5a514d3f6d8a6a871" member_type: VOTER last_known_addr { host: "127.14.103.61" port: 35785 } attrs { promote: false } } peers { permanent_uuid: "c42e0c000a9b4b53a34d41a257df494d" member_type: VOTER last_known_addr { host: "127.14.103.60" port: 40443 } attrs { promote: false } } }
I20260812 02:32:18.996795 22944 heartbeater.cc:464] Registering TS with master...
I20260812 02:32:18.996812 22970 sys_catalog.cc:458] T 00000000000000000000000000000000 P 295ba34a46a74756a017dbe2add34dbb [sys.catalog]: This master's current role is: LEADER
I20260812 02:32:18.997016 22944 heartbeater.cc:511] Master 127.14.103.62:37179 requested a full tablet report, sending...
I20260812 02:32:18.997766 21519 ts_manager.cc:194] Re-registered known tserver with Master: 9abf9de06df64b46b5db9181d885a31a (127.14.103.3:36861)
I20260812 02:32:18.997829 22213 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 198952c9f5644bc5a514d3f6d8a6a871 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, c42e0c000a9b4b53a34d41a257df494d (127.14.103.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "295ba34a46a74756a017dbe2add34dbb" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 37179 } } peers { permanent_uuid: "198952c9f5644bc5a514d3f6d8a6a871" member_type: VOTER last_known_addr { host: "127.14.103.61" port: 35785 } attrs { promote: false } } peers { permanent_uuid: "c42e0c000a9b4b53a34d41a257df494d" member_type: VOTER last_known_addr { host: "127.14.103.60" port: 40443 } attrs { promote: false } } }
I20260812 02:32:18.998623 21519 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.14.103.3@KRBTEST.COM'} at 127.14.103.3:57805
I20260812 02:32:18.992676 21520 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47244:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260812 02:32:18.999050 22948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:18.965933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.61:35785 (local address 127.14.103.3:52399)
0812 02:32:18.966693 (+ 760us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:18.966772 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0812 02:32:18.969746 (+ 2974us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:32:18.970116 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:32:18.970118 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:32:18.970298 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:32:18.972648 (+ 2350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:18.972655 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:18.973742 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:18.973745 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:18.975315 (+ 1570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:18.975326 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:18.977009 (+ 1683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:18.977015 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:18.977146 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:18.981476 (+ 4330us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:32:18.981496 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:32:18.983358 (+ 1862us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:32:18.997306 (+ 13948us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:32:18.997309 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:32:18.997312 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:32:18.997609 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:32:18.997935 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:32:18.997938 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:32:18.997940 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:32:18.998031 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:32:18.998474 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:32:18.998477 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:32:18.998567 (+ 90us) client_negotiation.cc:769] Sending connection context
0812 02:32:18.998659 (+ 92us) client_negotiation.cc:240] Negotiation successful
0812 02:32:18.998816 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":468,"mutex_wait_us":941,"thread_start_us":693,"threads_started":1}
I20260812 02:32:18.999576 22343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:18.965462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.3:52399 (local address 127.14.103.61:35785)
0812 02:32:18.965611 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:18.965616 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:18.966905 (+ 1289us) server_negotiation.cc:407] Connection header received
0812 02:32:18.969876 (+ 2971us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:18.969880 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:18.969932 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:18.969982 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:32:18.972812 (+ 2830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:18.973589 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:18.975773 (+ 2184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:18.976183 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:18.976816 (+ 633us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:18.983527 (+ 6711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:32:18.983531 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:32:18.983534 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:32:18.983565 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:32:18.996518 (+ 12953us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:18.997725 (+ 1207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:18.997729 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:18.997731 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:18.997821 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:18.998136 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:18.998139 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:18.998141 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:18.998351 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:32:18.999229 (+ 878us) server_negotiation.cc:1035] Waiting for connection context
0812 02:32:18.999311 (+ 82us) server_negotiation.cc:299] Negotiation successful
0812 02:32:18.999438 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260812 02:32:19.000370 22946 heartbeater.cc:347] Connected to a master server at 127.14.103.61:35785
I20260812 02:32:19.000478 22946 heartbeater.cc:464] Registering TS with master...
I20260812 02:32:19.000427 22972 sys_catalog.cc:455] T 00000000000000000000000000000000 P 198952c9f5644bc5a514d3f6d8a6a871 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "295ba34a46a74756a017dbe2add34dbb" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "295ba34a46a74756a017dbe2add34dbb" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 37179 } } peers { permanent_uuid: "198952c9f5644bc5a514d3f6d8a6a871" member_type: VOTER last_known_addr { host: "127.14.103.61" port: 35785 } attrs { promote: false } } peers { permanent_uuid: "c42e0c000a9b4b53a34d41a257df494d" member_type: VOTER last_known_addr { host: "127.14.103.60" port: 40443 } attrs { promote: false } } }
I20260812 02:32:19.000548 22972 sys_catalog.cc:458] T 00000000000000000000000000000000 P 198952c9f5644bc5a514d3f6d8a6a871 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 02:32:19.000701 22946 heartbeater.cc:511] Master 127.14.103.61:35785 requested a full tablet report, sending...
I20260812 02:32:19.001487 22194 ts_manager.cc:194] Registered new tserver with Master: 9abf9de06df64b46b5db9181d885a31a (127.14.103.3:36861)
W20260812 02:32:19.002528 21520 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 02:32:19.006364 22963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:18.969043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48914 (local address 127.14.103.60:40443)
0812 02:32:18.971091 (+ 2048us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:18.971099 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:18.971116 (+ 17us) server_negotiation.cc:407] Connection header received
0812 02:32:18.971164 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:18.971169 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:18.971227 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:18.971317 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:32:18.974379 (+ 3062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:18.975471 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:18.977570 (+ 2099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:18.977878 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:18.977981 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:18.981110 (+ 3129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:32:18.981117 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:32:18.981120 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:32:18.981150 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:32:19.003758 (+ 22608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:19.004628 (+ 870us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:19.004631 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:19.004632 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:19.004844 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:19.005277 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:19.005281 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:19.005283 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:19.005564 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:32:19.005708 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0812 02:32:19.006020 (+ 312us) server_negotiation.cc:299] Negotiation successful
0812 02:32:19.006192 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":1965,"thread_start_us":105,"threads_started":1}
I20260812 02:32:19.024670 22979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:19.017772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.2:35055 (local address 127.0.0.1:43582)
0812 02:32:19.018319 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:19.018337 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 02:32:19.018441 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:32:19.019983 (+ 1542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:32:19.019986 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:32:19.020023 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:32:19.020177 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:19.020184 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:19.021868 (+ 1684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:19.021871 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:19.022857 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:19.022868 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:19.024198 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:19.024202 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:19.024293 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:19.024320 (+ 27us) client_negotiation.cc:769] Sending connection context
0812 02:32:19.024386 (+ 66us) client_negotiation.cc:240] Negotiation successful
0812 02:32:19.024465 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":135,"threads_started":1}
I20260812 02:32:19.025094 22980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:19.017929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43582 (local address 127.14.103.2:35055)
0812 02:32:19.019346 (+ 1417us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:19.019369 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:19.019391 (+ 22us) server_negotiation.cc:407] Connection header received
0812 02:32:19.019462 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:19.019467 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:19.019662 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:19.019869 (+ 207us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:32:19.020360 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:19.021727 (+ 1367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:19.023022 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:19.024016 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:19.024166 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:19.024441 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0812 02:32:19.024559 (+ 118us) server_negotiation.cc:299] Negotiation successful
0812 02:32:19.024675 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1209,"thread_start_us":144,"threads_started":1}
I20260812 02:32:19.026439 22981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:19.018624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.1:46041 (local address 127.0.0.1:56066)
0812 02:32:19.020992 (+ 2368us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:19.021007 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 02:32:19.021097 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:32:19.021595 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:32:19.021598 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:32:19.021615 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:32:19.021728 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:19.021735 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:19.023088 (+ 1353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:19.023091 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:19.024478 (+ 1387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:19.024487 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:19.025908 (+ 1421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:19.025912 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:19.026079 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:19.026156 (+ 77us) client_negotiation.cc:769] Sending connection context
0812 02:32:19.026239 (+ 83us) client_negotiation.cc:240] Negotiation successful
0812 02:32:19.026322 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2282,"mutex_wait_us":135,"thread_start_us":91,"threads_started":1}
I20260812 02:32:19.026583 22984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:19.018107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.3:36861 (local address 127.0.0.1:41312)
0812 02:32:19.018916 (+ 809us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:19.018928 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 02:32:19.019118 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:32:19.020705 (+ 1587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:32:19.020708 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:32:19.020726 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:32:19.020845 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:19.020853 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:19.023088 (+ 2235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:19.023091 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:19.025216 (+ 2125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:19.025227 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:19.026075 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:19.026078 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:19.026227 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:19.026252 (+ 25us) client_negotiation.cc:769] Sending connection context
0812 02:32:19.026319 (+ 67us) client_negotiation.cc:240] Negotiation successful
0812 02:32:19.026400 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":756,"mutex_wait_us":121,"thread_start_us":101,"threads_started":1}
I20260812 02:32:19.026952 22985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:19.018857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56066 (local address 127.14.103.1:46041)
0812 02:32:19.019421 (+ 564us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:19.019426 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:19.021107 (+ 1681us) server_negotiation.cc:407] Connection header received
0812 02:32:19.021199 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:19.021204 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:19.021373 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:19.021879 (+ 506us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:32:19.022049 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:19.022904 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:19.024676 (+ 1772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:19.025589 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:19.025704 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:19.025840 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0812 02:32:19.026727 (+ 887us) server_negotiation.cc:299] Negotiation successful
0812 02:32:19.026851 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":104,"threads_started":1}
I20260812 02:32:19.027616 22982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:19.018496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41312 (local address 127.14.103.3:36861)
0812 02:32:19.020265 (+ 1769us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:19.020279 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:19.020302 (+ 23us) server_negotiation.cc:407] Connection header received
0812 02:32:19.020378 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:19.020383 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:19.020576 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:19.020729 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:32:19.021195 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:19.022101 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:19.025348 (+ 3247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:19.025927 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:19.026075 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:19.027294 (+ 1219us) server_negotiation.cc:1035] Waiting for connection context
0812 02:32:19.027350 (+ 56us) server_negotiation.cc:299] Negotiation successful
0812 02:32:19.027457 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1640,"thread_start_us":112,"threads_started":1}
I20260812 02:32:19.028712 22707 tablet_service.cc:1467] Processing CreateTablet for tablet a41546e21d9e409691167c1b80821f6d (DEFAULT_TABLE table=test-table [id=47352f6553c34a598e4932fe4e822805]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:32:19.029870 22707 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a41546e21d9e409691167c1b80821f6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:32:19.029738 22549 tablet_service.cc:1467] Processing CreateTablet for tablet a41546e21d9e409691167c1b80821f6d (DEFAULT_TABLE table=test-table [id=47352f6553c34a598e4932fe4e822805]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:32:19.031018 22549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a41546e21d9e409691167c1b80821f6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:32:19.038240 22986 tablet_bootstrap.cc:492] T a41546e21d9e409691167c1b80821f6d P 0294bdfc2c3e476fb7c39c969dee950b: Bootstrap starting.
I20260812 02:32:19.039248 22987 tablet_bootstrap.cc:492] T a41546e21d9e409691167c1b80821f6d P 88fa3b5bb4714fa691fcc87e8d327b8f: Bootstrap starting.
I20260812 02:32:19.039124 22865 tablet_service.cc:1467] Processing CreateTablet for tablet a41546e21d9e409691167c1b80821f6d (DEFAULT_TABLE table=test-table [id=47352f6553c34a598e4932fe4e822805]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:32:19.040791 22865 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a41546e21d9e409691167c1b80821f6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:32:19.041407 22987 tablet_bootstrap.cc:654] T a41546e21d9e409691167c1b80821f6d P 88fa3b5bb4714fa691fcc87e8d327b8f: Neither blocks nor log segments found. Creating new log.
I20260812 02:32:19.041453 22986 tablet_bootstrap.cc:654] T a41546e21d9e409691167c1b80821f6d P 0294bdfc2c3e476fb7c39c969dee950b: Neither blocks nor log segments found. Creating new log.
I20260812 02:32:19.043519 22986 log.cc:824] T a41546e21d9e409691167c1b80821f6d P 0294bdfc2c3e476fb7c39c969dee950b: Log is configured to *not* fsync() on all Append() calls
I20260812 02:32:19.043977 22987 log.cc:824] T a41546e21d9e409691167c1b80821f6d P 88fa3b5bb4714fa691fcc87e8d327b8f: Log is configured to *not* fsync() on all Append() calls
I20260812 02:32:19.046367 22986 tablet_bootstrap.cc:492] T a41546e21d9e409691167c1b80821f6d P 0294bdfc2c3e476fb7c39c969dee950b: No bootstrap required, opened a new log
I20260812 02:32:19.046638 22986 ts_tablet_manager.cc:1397] T a41546e21d9e409691167c1b80821f6d P 0294bdfc2c3e476fb7c39c969dee950b: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260812 02:32:19.051321 22987 tablet_bootstrap.cc:492] T a41546e21d9e409691167c1b80821f6d P 88fa3b5bb4714fa691fcc87e8d327b8f: No bootstrap required, opened a new log
I20260812 02:32:19.051584 22987 ts_tablet_manager.cc:1397] T a41546e21d9e409691167c1b80821f6d P 88fa3b5bb4714fa691fcc87e8d327b8f: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.000s
I20260812 02:32:19.052243 22986 raft_consensus.cc:348] T a41546e21d9e409691167c1b80821f6d P 0294bdfc2c3e476fb7c39c969dee950b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0294bdfc2c3e476fb7c39c969dee950b" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 35055 } } peers { permanent_uuid: "9abf9de06df64b46b5db9181d885a31a" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 36861 } } peers { permanent_uuid: "88fa3b5bb4714fa691fcc87e8d327b8f" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 46041 } }
I20260812 02:32:19.052572 22986 raft_consensus.cc:374] T a41546e21d9e409691167c1b80821f6d P 0294bdfc2c3e476fb7c39c969dee950b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:32:19.052644 22986 raft_consensus.cc:729] T a41546e21d9e409691167c1b80821f6d P 0294bdfc2c3e476fb7c39c969dee950b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0294bdfc2c3e476fb7c39c969dee950b, State: Initialized, Role: FOLLOWER
I20260812 02:32:19.053285 22990 tablet_bootstrap.cc:492] T a41546e21d9e409691167c1b80821f6d P 9abf9de06df64b46b5db9181d885a31a: Bootstrap starting.
I20260812 02:32:19.053285 22986 consensus_queue.cc:260] T a41546e21d9e409691167c1b80821f6d P 0294bdfc2c3e476fb7c39c969dee950b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0294bdfc2c3e476fb7c39c969dee950b" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 35055 } } peers { permanent_uuid: "9abf9de06df64b46b5db9181d885a31a" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 36861 } } peers { permanent_uuid: "88fa3b5bb4714fa691fcc87e8d327b8f" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 46041 } }
I20260812 02:32:19.054809 22986 ts_tablet_manager.cc:1428] T a41546e21d9e409691167c1b80821f6d P 0294bdfc2c3e476fb7c39c969dee950b: Time spent starting tablet: real 0.008s user 0.009s sys 0.000s
I20260812 02:32:19.056288 22987 raft_consensus.cc:348] T a41546e21d9e409691167c1b80821f6d P 88fa3b5bb4714fa691fcc87e8d327b8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0294bdfc2c3e476fb7c39c969dee950b" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 35055 } } peers { permanent_uuid: "9abf9de06df64b46b5db9181d885a31a" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 36861 } } peers { permanent_uuid: "88fa3b5bb4714fa691fcc87e8d327b8f" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 46041 } }
I20260812 02:32:19.056636 22987 raft_consensus.cc:374] T a41546e21d9e409691167c1b80821f6d P 88fa3b5bb4714fa691fcc87e8d327b8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:32:19.056718 22987 raft_consensus.cc:729] T a41546e21d9e409691167c1b80821f6d P 88fa3b5bb4714fa691fcc87e8d327b8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88fa3b5bb4714fa691fcc87e8d327b8f, State: Initialized, Role: FOLLOWER
I20260812 02:32:19.057305 22987 consensus_queue.cc:260] T a41546e21d9e409691167c1b80821f6d P 88fa3b5bb4714fa691fcc87e8d327b8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0294bdfc2c3e476fb7c39c969dee950b" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 35055 } } peers { permanent_uuid: "9abf9de06df64b46b5db9181d885a31a" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 36861 } } peers { permanent_uuid: "88fa3b5bb4714fa691fcc87e8d327b8f" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 46041 } }
I20260812 02:32:19.058254 22783 heartbeater.cc:503] Master 127.14.103.62:37179 was elected leader, sending a full tablet report...
I20260812 02:32:19.058580 22987 ts_tablet_manager.cc:1428] T a41546e21d9e409691167c1b80821f6d P 88fa3b5bb4714fa691fcc87e8d327b8f: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260812 02:32:19.058593 22623 heartbeater.cc:503] Master 127.14.103.62:37179 was elected leader, sending a full tablet report...
I20260812 02:32:19.059409 22990 tablet_bootstrap.cc:654] T a41546e21d9e409691167c1b80821f6d P 9abf9de06df64b46b5db9181d885a31a: Neither blocks nor log segments found. Creating new log.
I20260812 02:32:19.060853 22990 log.cc:824] T a41546e21d9e409691167c1b80821f6d P 9abf9de06df64b46b5db9181d885a31a: Log is configured to *not* fsync() on all Append() calls
I20260812 02:32:19.063638 22990 tablet_bootstrap.cc:492] T a41546e21d9e409691167c1b80821f6d P 9abf9de06df64b46b5db9181d885a31a: No bootstrap required, opened a new log
I20260812 02:32:19.063980 22990 ts_tablet_manager.cc:1397] T a41546e21d9e409691167c1b80821f6d P 9abf9de06df64b46b5db9181d885a31a: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260812 02:32:19.068854 22990 raft_consensus.cc:348] T a41546e21d9e409691167c1b80821f6d P 9abf9de06df64b46b5db9181d885a31a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0294bdfc2c3e476fb7c39c969dee950b" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 35055 } } peers { permanent_uuid: "9abf9de06df64b46b5db9181d885a31a" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 36861 } } peers { permanent_uuid: "88fa3b5bb4714fa691fcc87e8d327b8f" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 46041 } }
I20260812 02:32:19.069175 22990 raft_consensus.cc:374] T a41546e21d9e409691167c1b80821f6d P 9abf9de06df64b46b5db9181d885a31a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:32:19.069216 22990 raft_consensus.cc:729] T a41546e21d9e409691167c1b80821f6d P 9abf9de06df64b46b5db9181d885a31a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9abf9de06df64b46b5db9181d885a31a, State: Initialized, Role: FOLLOWER
I20260812 02:32:19.069639 22990 consensus_queue.cc:260] T a41546e21d9e409691167c1b80821f6d P 9abf9de06df64b46b5db9181d885a31a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0294bdfc2c3e476fb7c39c969dee950b" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 35055 } } peers { permanent_uuid: "9abf9de06df64b46b5db9181d885a31a" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 36861 } } peers { permanent_uuid: "88fa3b5bb4714fa691fcc87e8d327b8f" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 46041 } }
I20260812 02:32:19.070763 22990 ts_tablet_manager.cc:1428] T a41546e21d9e409691167c1b80821f6d P 9abf9de06df64b46b5db9181d885a31a: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260812 02:32:19.070619 22944 heartbeater.cc:503] Master 127.14.103.62:37179 was elected leader, sending a full tablet report...
W20260812 02:32:19.110276 22626 tablet.cc:2369] T a41546e21d9e409691167c1b80821f6d P 88fa3b5bb4714fa691fcc87e8d327b8f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 02:32:19.155054 22991 raft_consensus.cc:482] T a41546e21d9e409691167c1b80821f6d P 0294bdfc2c3e476fb7c39c969dee950b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 02:32:19.155284 22991 raft_consensus.cc:504] T a41546e21d9e409691167c1b80821f6d P 0294bdfc2c3e476fb7c39c969dee950b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0294bdfc2c3e476fb7c39c969dee950b" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 35055 } } peers { permanent_uuid: "9abf9de06df64b46b5db9181d885a31a" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 36861 } } peers { permanent_uuid: "88fa3b5bb4714fa691fcc87e8d327b8f" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 46041 } }
I20260812 02:32:19.156488 22991 leader_election.cc:288] T a41546e21d9e409691167c1b80821f6d P 0294bdfc2c3e476fb7c39c969dee950b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9abf9de06df64b46b5db9181d885a31a (127.14.103.3:36861), 88fa3b5bb4714fa691fcc87e8d327b8f (127.14.103.1:46041)
W20260812 02:32:19.158708 22787 tablet.cc:2369] T a41546e21d9e409691167c1b80821f6d P 0294bdfc2c3e476fb7c39c969dee950b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 02:32:19.162815 22985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:19.157141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.2:38201 (local address 127.14.103.1:46041)
0812 02:32:19.157285 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:19.157290 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:19.157306 (+ 16us) server_negotiation.cc:407] Connection header received
0812 02:32:19.157372 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:19.157374 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:19.157412 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:19.157507 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:32:19.157921 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:19.158736 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:19.161034 (+ 2298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:19.161835 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:19.161997 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:19.162053 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0812 02:32:19.162485 (+ 432us) server_negotiation.cc:299] Negotiation successful
0812 02:32:19.162670 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260812 02:32:19.162815 22982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:19.157237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.2:46263 (local address 127.14.103.3:36861)
0812 02:32:19.157379 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:19.157382 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:19.157394 (+ 12us) server_negotiation.cc:407] Connection header received
0812 02:32:19.157440 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:19.157442 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:19.157480 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:19.157579 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:32:19.158333 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:19.159367 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:19.161037 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:19.161861 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:19.161997 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:19.162053 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0812 02:32:19.162482 (+ 429us) server_negotiation.cc:299] Negotiation successful
0812 02:32:19.162670 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260812 02:32:19.162994 22791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:19.156854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.3:36861 (local address 127.14.103.2:46263)
0812 02:32:19.157083 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:19.157101 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 02:32:19.157220 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:32:19.157592 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:32:19.157595 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:32:19.157612 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:32:19.157983 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:19.158000 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:19.159513 (+ 1513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:19.159517 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:19.160845 (+ 1328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:19.160856 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:19.162050 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:19.162054 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:19.162267 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:19.162311 (+ 44us) client_negotiation.cc:769] Sending connection context
0812 02:32:19.162403 (+ 92us) client_negotiation.cc:240] Negotiation successful
0812 02:32:19.162531 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"mutex_wait_us":191,"spinlock_wait_cycles":5888}
I20260812 02:32:19.162998 22789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:19.156819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.1:46041 (local address 127.14.103.2:38201)
0812 02:32:19.157079 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:19.157097 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 02:32:19.157217 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:32:19.157521 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:32:19.157524 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:32:19.157541 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:32:19.157784 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:19.157792 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:19.159272 (+ 1480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:19.159276 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:19.160777 (+ 1501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:19.160794 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:19.162050 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:19.162054 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:19.162274 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:19.162318 (+ 44us) client_negotiation.cc:769] Sending connection context
0812 02:32:19.162424 (+ 106us) client_negotiation.cc:240] Negotiation successful
0812 02:32:19.162541 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"mutex_wait_us":104,"spinlock_wait_cycles":5376}
I20260812 02:32:19.163928 22887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a41546e21d9e409691167c1b80821f6d" candidate_uuid: "0294bdfc2c3e476fb7c39c969dee950b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9abf9de06df64b46b5db9181d885a31a" is_pre_election: true
I20260812 02:32:19.163928 22569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a41546e21d9e409691167c1b80821f6d" candidate_uuid: "0294bdfc2c3e476fb7c39c969dee950b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88fa3b5bb4714fa691fcc87e8d327b8f" is_pre_election: true
I20260812 02:32:19.164299 22569 raft_consensus.cc:2454] T a41546e21d9e409691167c1b80821f6d P 88fa3b5bb4714fa691fcc87e8d327b8f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0294bdfc2c3e476fb7c39c969dee950b in term 0.
I20260812 02:32:19.164299 22887 raft_consensus.cc:2454] T a41546e21d9e409691167c1b80821f6d P 9abf9de06df64b46b5db9181d885a31a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0294bdfc2c3e476fb7c39c969dee950b in term 0.
I20260812 02:32:19.164917 22655 leader_election.cc:302] T a41546e21d9e409691167c1b80821f6d P 0294bdfc2c3e476fb7c39c969dee950b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0294bdfc2c3e476fb7c39c969dee950b, 88fa3b5bb4714fa691fcc87e8d327b8f; no voters:
I20260812 02:32:19.165287 22991 raft_consensus.cc:2781] T a41546e21d9e409691167c1b80821f6d P 0294bdfc2c3e476fb7c39c969dee950b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 02:32:19.165374 22991 raft_consensus.cc:482] T a41546e21d9e409691167c1b80821f6d P 0294bdfc2c3e476fb7c39c969dee950b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 02:32:19.165452 22991 raft_consensus.cc:3037] T a41546e21d9e409691167c1b80821f6d P 0294bdfc2c3e476fb7c39c969dee950b [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:32:19.166837 22991 raft_consensus.cc:504] T a41546e21d9e409691167c1b80821f6d P 0294bdfc2c3e476fb7c39c969dee950b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0294bdfc2c3e476fb7c39c969dee950b" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 35055 } } peers { permanent_uuid: "9abf9de06df64b46b5db9181d885a31a" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 36861 } } peers { permanent_uuid: "88fa3b5bb4714fa691fcc87e8d327b8f" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 46041 } }
I20260812 02:32:19.167376 22991 leader_election.cc:288] T a41546e21d9e409691167c1b80821f6d P 0294bdfc2c3e476fb7c39c969dee950b [CANDIDATE]: Term 1 election: Requested vote from peers 9abf9de06df64b46b5db9181d885a31a (127.14.103.3:36861), 88fa3b5bb4714fa691fcc87e8d327b8f (127.14.103.1:46041)
I20260812 02:32:19.167888 22887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a41546e21d9e409691167c1b80821f6d" candidate_uuid: "0294bdfc2c3e476fb7c39c969dee950b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9abf9de06df64b46b5db9181d885a31a"
I20260812 02:32:19.168079 22887 raft_consensus.cc:3037] T a41546e21d9e409691167c1b80821f6d P 9abf9de06df64b46b5db9181d885a31a [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:32:19.168318 22569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a41546e21d9e409691167c1b80821f6d" candidate_uuid: "0294bdfc2c3e476fb7c39c969dee950b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88fa3b5bb4714fa691fcc87e8d327b8f"
I20260812 02:32:19.168485 22569 raft_consensus.cc:3037] T a41546e21d9e409691167c1b80821f6d P 88fa3b5bb4714fa691fcc87e8d327b8f [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:32:19.169471 22887 raft_consensus.cc:2454] T a41546e21d9e409691167c1b80821f6d P 9abf9de06df64b46b5db9181d885a31a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0294bdfc2c3e476fb7c39c969dee950b in term 1.
I20260812 02:32:19.169900 22652 leader_election.cc:302] T a41546e21d9e409691167c1b80821f6d P 0294bdfc2c3e476fb7c39c969dee950b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0294bdfc2c3e476fb7c39c969dee950b, 9abf9de06df64b46b5db9181d885a31a; no voters:
I20260812 02:32:19.170032 22569 raft_consensus.cc:2454] T a41546e21d9e409691167c1b80821f6d P 88fa3b5bb4714fa691fcc87e8d327b8f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0294bdfc2c3e476fb7c39c969dee950b in term 1.
I20260812 02:32:19.170114 22991 raft_consensus.cc:2781] T a41546e21d9e409691167c1b80821f6d P 0294bdfc2c3e476fb7c39c969dee950b [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:32:19.170387 22991 raft_consensus.cc:686] T a41546e21d9e409691167c1b80821f6d P 0294bdfc2c3e476fb7c39c969dee950b [term 1 LEADER]: Becoming Leader. State: Replica: 0294bdfc2c3e476fb7c39c969dee950b, State: Running, Role: LEADER
I20260812 02:32:19.170807 22991 consensus_queue.cc:237] T a41546e21d9e409691167c1b80821f6d P 0294bdfc2c3e476fb7c39c969dee950b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0294bdfc2c3e476fb7c39c969dee950b" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 35055 } } peers { permanent_uuid: "9abf9de06df64b46b5db9181d885a31a" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 36861 } } peers { permanent_uuid: "88fa3b5bb4714fa691fcc87e8d327b8f" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 46041 } }
I20260812 02:32:19.175307 21519 catalog_manager.cc:5495] T a41546e21d9e409691167c1b80821f6d P 0294bdfc2c3e476fb7c39c969dee950b reported cstate change: term changed from 0 to 1, leader changed from <none> to 0294bdfc2c3e476fb7c39c969dee950b (127.14.103.2). New cstate: current_term: 1 leader_uuid: "0294bdfc2c3e476fb7c39c969dee950b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0294bdfc2c3e476fb7c39c969dee950b" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 35055 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9abf9de06df64b46b5db9181d885a31a" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 36861 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88fa3b5bb4714fa691fcc87e8d327b8f" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 46041 } health_report { overall_health: UNKNOWN } } }
I20260812 02:32:19.195559 22980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:19.188734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43596 (local address 127.14.103.2:35055)
0812 02:32:19.188901 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:19.188904 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:19.189015 (+ 111us) server_negotiation.cc:407] Connection header received
0812 02:32:19.189163 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:19.189166 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:19.189228 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:19.189334 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 02:32:19.190193 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:19.190809 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:19.192044 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:19.192301 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:19.192408 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:19.193034 (+ 626us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 02:32:19.194182 (+ 1148us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 02:32:19.194401 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0812 02:32:19.194769 (+ 368us) server_negotiation.cc:299] Negotiation successful
0812 02:32:19.195264 (+ 495us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260812 02:32:19.203761 22887 raft_consensus.cc:1261] T a41546e21d9e409691167c1b80821f6d P 9abf9de06df64b46b5db9181d885a31a [term 1 FOLLOWER]: Refusing update from remote peer 0294bdfc2c3e476fb7c39c969dee950b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:32:19.203761 22569 raft_consensus.cc:1261] T a41546e21d9e409691167c1b80821f6d P 88fa3b5bb4714fa691fcc87e8d327b8f [term 1 FOLLOWER]: Refusing update from remote peer 0294bdfc2c3e476fb7c39c969dee950b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:32:19.204689 22991 consensus_queue.cc:1035] T a41546e21d9e409691167c1b80821f6d P 0294bdfc2c3e476fb7c39c969dee950b [LEADER]: Connected to new peer: Peer: permanent_uuid: "88fa3b5bb4714fa691fcc87e8d327b8f" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 46041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:32:19.205054 22996 consensus_queue.cc:1035] T a41546e21d9e409691167c1b80821f6d P 0294bdfc2c3e476fb7c39c969dee950b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9abf9de06df64b46b5db9181d885a31a" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 36861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260812 02:32:19.214720 22947 tablet.cc:2369] T a41546e21d9e409691167c1b80821f6d P 9abf9de06df64b46b5db9181d885a31a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 02:32:19.216085 23001 mvcc.cc:205] Tried to move back new op lower bound from 7317511942968860672 to 7317511942847008768. Current Snapshot: MvccSnapshot[applied={T|T < 7317511942968860672}]
I20260812 02:32:19.216156 23002 mvcc.cc:205] Tried to move back new op lower bound from 7317511942968860672 to 7317511942847008768. Current Snapshot: MvccSnapshot[applied={T|T < 7317511942968860672}]
I20260812 02:32:19.231355 21520 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47244:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 02:32:19.231721 21520 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47244:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 02:32:19.241134 21520 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 295ba34a46a74756a017dbe2add34dbb: Sending DeleteTablet for 3 replicas of tablet a41546e21d9e409691167c1b80821f6d
I20260812 02:32:19.242054 22549 tablet_service.cc:1514] Processing DeleteTablet for tablet a41546e21d9e409691167c1b80821f6d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:32:19 UTC) from {username='oryx', principal='oryx/127.14.103.62@KRBTEST.COM'} at 127.0.0.1:56066
I20260812 02:32:19.242229 22865 tablet_service.cc:1514] Processing DeleteTablet for tablet a41546e21d9e409691167c1b80821f6d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:32:19 UTC) from {username='oryx', principal='oryx/127.14.103.62@KRBTEST.COM'} at 127.0.0.1:41312
I20260812 02:32:19.242880 23009 tablet_replica.cc:330] T a41546e21d9e409691167c1b80821f6d P 88fa3b5bb4714fa691fcc87e8d327b8f: stopping tablet replica
I20260812 02:32:19.243295 23009 raft_consensus.cc:2229] T a41546e21d9e409691167c1b80821f6d P 88fa3b5bb4714fa691fcc87e8d327b8f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 02:32:19.243655 23009 raft_consensus.cc:2258] T a41546e21d9e409691167c1b80821f6d P 88fa3b5bb4714fa691fcc87e8d327b8f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:32:19.243779 22707 tablet_service.cc:1514] Processing DeleteTablet for tablet a41546e21d9e409691167c1b80821f6d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:32:19 UTC) from {username='oryx', principal='oryx/127.14.103.62@KRBTEST.COM'} at 127.0.0.1:43582
I20260812 02:32:19.244074 23010 tablet_replica.cc:330] T a41546e21d9e409691167c1b80821f6d P 9abf9de06df64b46b5db9181d885a31a: stopping tablet replica
I20260812 02:32:19.244438 23010 raft_consensus.cc:2229] T a41546e21d9e409691167c1b80821f6d P 9abf9de06df64b46b5db9181d885a31a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 02:32:19.244717 23010 raft_consensus.cc:2258] T a41546e21d9e409691167c1b80821f6d P 9abf9de06df64b46b5db9181d885a31a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:32:19.246610 14748 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbep4M4/build/debug/bin/kudu with pid 21484
I20260812 02:32:19.254721 23009 ts_tablet_manager.cc:1905] T a41546e21d9e409691167c1b80821f6d P 88fa3b5bb4714fa691fcc87e8d327b8f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:32:19.257794 23009 ts_tablet_manager.cc:1918] T a41546e21d9e409691167c1b80821f6d P 88fa3b5bb4714fa691fcc87e8d327b8f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 02:32:19.258072 23009 log.cc:1196] T a41546e21d9e409691167c1b80821f6d P 88fa3b5bb4714fa691fcc87e8d327b8f: Deleting WAL directory at /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/minicluster-data/ts-0/wal/wals/a41546e21d9e409691167c1b80821f6d
I20260812 02:32:19.258776 23009 ts_tablet_manager.cc:1939] T a41546e21d9e409691167c1b80821f6d P 88fa3b5bb4714fa691fcc87e8d327b8f: Deleting consensus metadata
I20260812 02:32:19.258787 23010 ts_tablet_manager.cc:1905] T a41546e21d9e409691167c1b80821f6d P 9abf9de06df64b46b5db9181d885a31a: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260812 02:32:19.260561 22494 connection.cc:441] server connection from 127.0.0.1:56066 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56066 (request call id 1) could send its response
I20260812 02:32:19.262519 23010 ts_tablet_manager.cc:1918] T a41546e21d9e409691167c1b80821f6d P 9abf9de06df64b46b5db9181d885a31a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 02:32:19.262656 23010 log.cc:1196] T a41546e21d9e409691167c1b80821f6d P 9abf9de06df64b46b5db9181d885a31a: Deleting WAL directory at /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/minicluster-data/ts-2/wal/wals/a41546e21d9e409691167c1b80821f6d
I20260812 02:32:19.263082 23010 ts_tablet_manager.cc:1939] T a41546e21d9e409691167c1b80821f6d P 9abf9de06df64b46b5db9181d885a31a: Deleting consensus metadata
I20260812 02:32:19.264515 23011 tablet_replica.cc:330] T a41546e21d9e409691167c1b80821f6d P 0294bdfc2c3e476fb7c39c969dee950b: stopping tablet replica
W20260812 02:32:19.264493 22811 connection.cc:441] server connection from 127.0.0.1:41312 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41312 (request call id 1) could send its response
I20260812 02:32:19.264878 23011 raft_consensus.cc:2229] T a41546e21d9e409691167c1b80821f6d P 0294bdfc2c3e476fb7c39c969dee950b [term 1 LEADER]: Raft consensus shutting down.
I20260812 02:32:19.265372 23011 raft_consensus.cc:2258] T a41546e21d9e409691167c1b80821f6d P 0294bdfc2c3e476fb7c39c969dee950b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:32:19.271266 23011 ts_tablet_manager.cc:1905] T a41546e21d9e409691167c1b80821f6d P 0294bdfc2c3e476fb7c39c969dee950b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:32:19.279152 23011 ts_tablet_manager.cc:1918] T a41546e21d9e409691167c1b80821f6d P 0294bdfc2c3e476fb7c39c969dee950b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 02:32:19.279311 23011 log.cc:1196] T a41546e21d9e409691167c1b80821f6d P 0294bdfc2c3e476fb7c39c969dee950b: Deleting WAL directory at /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/minicluster-data/ts-1/wal/wals/a41546e21d9e409691167c1b80821f6d
I20260812 02:32:19.280160 23011 ts_tablet_manager.cc:1939] T a41546e21d9e409691167c1b80821f6d P 0294bdfc2c3e476fb7c39c969dee950b: Deleting consensus metadata
W20260812 02:32:19.282765 22655 connection.cc:441] server connection from 127.0.0.1:43582 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43582 (request call id 1) could send its response
I20260812 02:32:19.284222 22343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:19.246374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35132 (local address 127.14.103.61:35785)
0812 02:32:19.246636 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:19.246641 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:19.262828 (+ 16187us) server_negotiation.cc:407] Connection header received
0812 02:32:19.263096 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:19.263100 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:19.263157 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:19.263213 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:32:19.265232 (+ 2019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:19.266075 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:19.273841 (+ 7766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:19.274198 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:19.274295 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:19.276782 (+ 2487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:32:19.276787 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:32:19.276790 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:32:19.276837 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:32:19.280751 (+ 3914us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:19.282479 (+ 1728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:19.282483 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:19.282485 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:19.282578 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:19.282936 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:19.282940 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:19.282942 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:19.283390 (+ 448us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:32:19.283516 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0812 02:32:19.283901 (+ 385us) server_negotiation.cc:299] Negotiation successful
0812 02:32:19.284050 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166}
I20260812 02:32:19.285452 22630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:19.266998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48930 (local address 127.14.103.60:40443)
0812 02:32:19.267122 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:19.267126 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:19.267884 (+ 758us) server_negotiation.cc:407] Connection header received
0812 02:32:19.268076 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:19.268079 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:19.268135 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:19.268190 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:32:19.270287 (+ 2097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:19.271160 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:19.272998 (+ 1838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:19.273359 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:19.273443 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:19.278410 (+ 4967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:32:19.278422 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:32:19.278425 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:32:19.278471 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:32:19.282150 (+ 3679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:19.282713 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:19.282720 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:19.282723 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:19.282825 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:19.283345 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:19.283351 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:19.283356 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:19.283780 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:32:19.283894 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0812 02:32:19.285018 (+ 1124us) server_negotiation.cc:299] Negotiation successful
0812 02:32:19.285284 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260812 02:32:19.285677 22959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:19.265980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35134 (local address 127.14.103.61:35785)
0812 02:32:19.266165 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:19.266170 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:19.266516 (+ 346us) server_negotiation.cc:407] Connection header received
0812 02:32:19.266733 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:19.266736 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:19.266788 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:19.266846 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:32:19.267965 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:19.270252 (+ 2287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:19.272088 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:19.272688 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:19.272813 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:19.279969 (+ 7156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:32:19.279980 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:32:19.279987 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:32:19.280022 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:32:19.282051 (+ 2029us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:19.283428 (+ 1377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:19.283431 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:19.283433 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:19.283695 (+ 262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:19.284261 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:19.284266 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:19.284271 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:19.284646 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:32:19.284786 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0812 02:32:19.285254 (+ 468us) server_negotiation.cc:299] Negotiation successful
0812 02:32:19.285388 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260812 02:32:19.298431 22963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:19.246490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48926 (local address 127.14.103.60:40443)
0812 02:32:19.246742 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:19.246749 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:19.285752 (+ 39003us) server_negotiation.cc:407] Connection header received
0812 02:32:19.285919 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:19.285922 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:19.285979 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:19.286031 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:32:19.287233 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:19.288078 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:19.290282 (+ 2204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:19.290587 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:19.290698 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:19.294077 (+ 3379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:32:19.294084 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:32:19.294088 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:32:19.294141 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:32:19.296343 (+ 2202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:19.296957 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:19.296961 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:19.296965 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:19.297083 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:19.297424 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:19.297427 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:19.297429 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:19.297690 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:32:19.297843 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0812 02:32:19.298101 (+ 258us) server_negotiation.cc:299] Negotiation successful
0812 02:32:19.298245 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169}
I20260812 02:32:20.221012 23021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:20.220373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:37179 (local address 127.14.103.3:54179)
0812 02:32:20.220723 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:20.220840 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.62:37179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":178,"threads_started":1}
W20260812 02:32:20.221853 22944 heartbeater.cc:644] Failed to heartbeat to 127.14.103.62:37179 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.14.103.62:37179: connect: Connection refused (error 111)
I20260812 02:32:20.223716 23022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:20.222898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:37179 (local address 127.14.103.1:40677)
0812 02:32:20.223540 (+ 642us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:20.223614 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.62:37179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":577,"thread_start_us":286,"threads_started":1}
I20260812 02:32:20.223716 23024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:20.223258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:37179 (local address 127.14.103.2:50651)
0812 02:32:20.223510 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:20.223613 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.62:37179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":79,"threads_started":1}
W20260812 02:32:20.224041 22783 heartbeater.cc:644] Failed to heartbeat to 127.14.103.62:37179 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.14.103.62:37179: connect: Connection refused (error 111)
W20260812 02:32:20.224305 22623 heartbeater.cc:644] Failed to heartbeat to 127.14.103.62:37179 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.14.103.62:37179: connect: Connection refused (error 111)
I20260812 02:32:20.740386 23025 raft_consensus.cc:482] T 00000000000000000000000000000000 P c42e0c000a9b4b53a34d41a257df494d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 295ba34a46a74756a017dbe2add34dbb)
I20260812 02:32:20.740764 23025 raft_consensus.cc:504] T 00000000000000000000000000000000 P c42e0c000a9b4b53a34d41a257df494d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "295ba34a46a74756a017dbe2add34dbb" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 37179 } } peers { permanent_uuid: "198952c9f5644bc5a514d3f6d8a6a871" member_type: VOTER last_known_addr { host: "127.14.103.61" port: 35785 } attrs { promote: false } } peers { permanent_uuid: "c42e0c000a9b4b53a34d41a257df494d" member_type: VOTER last_known_addr { host: "127.14.103.60" port: 40443 } attrs { promote: false } }
I20260812 02:32:20.741174 23026 raft_consensus.cc:482] T 00000000000000000000000000000000 P 198952c9f5644bc5a514d3f6d8a6a871 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 295ba34a46a74756a017dbe2add34dbb)
I20260812 02:32:20.741434 23026 raft_consensus.cc:504] T 00000000000000000000000000000000 P 198952c9f5644bc5a514d3f6d8a6a871 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "295ba34a46a74756a017dbe2add34dbb" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 37179 } } peers { permanent_uuid: "198952c9f5644bc5a514d3f6d8a6a871" member_type: VOTER last_known_addr { host: "127.14.103.61" port: 35785 } attrs { promote: false } } peers { permanent_uuid: "c42e0c000a9b4b53a34d41a257df494d" member_type: VOTER last_known_addr { host: "127.14.103.60" port: 40443 } attrs { promote: false } }
I20260812 02:32:20.742015 23026 leader_election.cc:288] T 00000000000000000000000000000000 P 198952c9f5644bc5a514d3f6d8a6a871 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 295ba34a46a74756a017dbe2add34dbb (127.14.103.62:37179), c42e0c000a9b4b53a34d41a257df494d (127.14.103.60:40443)
I20260812 02:32:20.742018 23025 leader_election.cc:288] T 00000000000000000000000000000000 P c42e0c000a9b4b53a34d41a257df494d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 295ba34a46a74756a017dbe2add34dbb (127.14.103.62:37179), 198952c9f5644bc5a514d3f6d8a6a871 (127.14.103.61:35785)
I20260812 02:32:20.742799 23028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:20.742221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:37179 (local address 127.0.0.1:47690)
0812 02:32:20.742584 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:20.742677 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.62:37179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":82,"threads_started":1}
I20260812 02:32:20.742862 23027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:20.742215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:37179 (local address 127.0.0.1:47700)
0812 02:32:20.742704 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:20.742763 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.62:37179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":69,"threads_started":1}
W20260812 02:32:20.743010 22396 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.103.62:37179: connect: Connection refused (error 111)
W20260812 02:32:20.743125 22183 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.103.62:37179: connect: Connection refused (error 111)
I20260812 02:32:20.744436 23028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:20.744188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:37179 (local address 127.0.0.1:47716)
0812 02:32:20.744296 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:20.744357 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.62:37179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260812 02:32:20.744473 23031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:20.743951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:37179 (local address 127.0.0.1:47710)
0812 02:32:20.744357 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:20.744407 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.62:37179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260812 02:32:20.744562 22396 leader_election.cc:334] T 00000000000000000000000000000000 P c42e0c000a9b4b53a34d41a257df494d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 295ba34a46a74756a017dbe2add34dbb (127.14.103.62:37179): Network error: Client connection negotiation failed: client connection to 127.14.103.62:37179: connect: Connection refused (error 111)
W20260812 02:32:20.744793 22183 leader_election.cc:334] T 00000000000000000000000000000000 P 198952c9f5644bc5a514d3f6d8a6a871 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 295ba34a46a74756a017dbe2add34dbb (127.14.103.62:37179): Network error: Client connection negotiation failed: client connection to 127.14.103.62:37179: connect: Connection refused (error 111)
I20260812 02:32:20.747346 23030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:20.742382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35150 (local address 127.14.103.61:35785)
0812 02:32:20.742610 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:20.742615 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:20.742727 (+ 112us) server_negotiation.cc:407] Connection header received
0812 02:32:20.743400 (+ 673us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:20.743404 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:20.743461 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:20.743620 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:32:20.744228 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:20.745100 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:20.746265 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:20.746691 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:20.746824 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:20.746898 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0812 02:32:20.747138 (+ 240us) server_negotiation.cc:299] Negotiation successful
0812 02:32:20.747213 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":59,"threads_started":1}
I20260812 02:32:20.747347 23029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:20.742318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.61:35785 (local address 127.0.0.1:35150)
0812 02:32:20.742637 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:20.742653 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 02:32:20.742815 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:32:20.743819 (+ 1004us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:32:20.743825 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:32:20.743850 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:32:20.743962 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:20.743972 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:20.745232 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:20.745235 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:20.746117 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:20.746127 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:20.746829 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:20.746833 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:20.746946 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:20.747022 (+ 76us) client_negotiation.cc:769] Sending connection context
0812 02:32:20.747122 (+ 100us) client_negotiation.cc:240] Negotiation successful
0812 02:32:20.747205 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"mutex_wait_us":1,"thread_start_us":80,"threads_started":1}
I20260812 02:32:20.747529 23032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:20.742872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48938 (local address 127.14.103.60:40443)
0812 02:32:20.744255 (+ 1383us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:20.744260 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:20.744278 (+ 18us) server_negotiation.cc:407] Connection header received
0812 02:32:20.744319 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:20.744321 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:20.744363 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:20.744698 (+ 335us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:32:20.744852 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:20.745480 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:20.746510 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:20.746946 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:20.747103 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:20.747166 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0812 02:32:20.747352 (+ 186us) server_negotiation.cc:299] Negotiation successful
0812 02:32:20.747424 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":875,"thread_start_us":92,"threads_started":1}
I20260812 02:32:20.747527 23027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:20.742551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.60:40443 (local address 127.0.0.1:48938)
0812 02:32:20.743028 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:20.743042 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 02:32:20.743323 (+ 281us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:32:20.744500 (+ 1177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:32:20.744503 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:32:20.744521 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:32:20.744600 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:20.744605 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:20.745593 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:20.745598 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:20.746383 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:20.746391 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:20.747113 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:20.747116 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:20.747209 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:20.747237 (+ 28us) client_negotiation.cc:769] Sending connection context
0812 02:32:20.747325 (+ 88us) client_negotiation.cc:240] Negotiation successful
0812 02:32:20.747400 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":46,"threads_started":1}
I20260812 02:32:20.748008 22213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c42e0c000a9b4b53a34d41a257df494d" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "198952c9f5644bc5a514d3f6d8a6a871" is_pre_election: true
I20260812 02:32:20.748090 22425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "198952c9f5644bc5a514d3f6d8a6a871" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "c42e0c000a9b4b53a34d41a257df494d" is_pre_election: true
I20260812 02:32:20.748278 22213 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 198952c9f5644bc5a514d3f6d8a6a871 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c42e0c000a9b4b53a34d41a257df494d in term 1.
I20260812 02:32:20.748315 22425 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c42e0c000a9b4b53a34d41a257df494d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 198952c9f5644bc5a514d3f6d8a6a871 in term 1.
I20260812 02:32:20.748657 22183 leader_election.cc:302] T 00000000000000000000000000000000 P 198952c9f5644bc5a514d3f6d8a6a871 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 198952c9f5644bc5a514d3f6d8a6a871, c42e0c000a9b4b53a34d41a257df494d; no voters: 295ba34a46a74756a017dbe2add34dbb
I20260812 02:32:20.748690 22397 leader_election.cc:302] T 00000000000000000000000000000000 P c42e0c000a9b4b53a34d41a257df494d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 198952c9f5644bc5a514d3f6d8a6a871, c42e0c000a9b4b53a34d41a257df494d; no voters: 295ba34a46a74756a017dbe2add34dbb
I20260812 02:32:20.748929 23026 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 198952c9f5644bc5a514d3f6d8a6a871 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260812 02:32:20.748953 23025 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c42e0c000a9b4b53a34d41a257df494d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260812 02:32:20.748998 23026 raft_consensus.cc:482] T 00000000000000000000000000000000 P 198952c9f5644bc5a514d3f6d8a6a871 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 295ba34a46a74756a017dbe2add34dbb)
I20260812 02:32:20.749058 23026 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 198952c9f5644bc5a514d3f6d8a6a871 [term 1 FOLLOWER]: Advancing to term 2
I20260812 02:32:20.749053 23025 raft_consensus.cc:482] T 00000000000000000000000000000000 P c42e0c000a9b4b53a34d41a257df494d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 295ba34a46a74756a017dbe2add34dbb)
I20260812 02:32:20.749104 23025 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c42e0c000a9b4b53a34d41a257df494d [term 1 FOLLOWER]: Advancing to term 2
I20260812 02:32:20.750192 23025 raft_consensus.cc:504] T 00000000000000000000000000000000 P c42e0c000a9b4b53a34d41a257df494d [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "295ba34a46a74756a017dbe2add34dbb" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 37179 } } peers { permanent_uuid: "198952c9f5644bc5a514d3f6d8a6a871" member_type: VOTER last_known_addr { host: "127.14.103.61" port: 35785 } attrs { promote: false } } peers { permanent_uuid: "c42e0c000a9b4b53a34d41a257df494d" member_type: VOTER last_known_addr { host: "127.14.103.60" port: 40443 } attrs { promote: false } }
I20260812 02:32:20.750221 23026 raft_consensus.cc:504] T 00000000000000000000000000000000 P 198952c9f5644bc5a514d3f6d8a6a871 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "295ba34a46a74756a017dbe2add34dbb" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 37179 } } peers { permanent_uuid: "198952c9f5644bc5a514d3f6d8a6a871" member_type: VOTER last_known_addr { host: "127.14.103.61" port: 35785 } attrs { promote: false } } peers { permanent_uuid: "c42e0c000a9b4b53a34d41a257df494d" member_type: VOTER last_known_addr { host: "127.14.103.60" port: 40443 } attrs { promote: false } }
I20260812 02:32:20.750624 23026 leader_election.cc:288] T 00000000000000000000000000000000 P 198952c9f5644bc5a514d3f6d8a6a871 [CANDIDATE]: Term 2 election: Requested vote from peers 295ba34a46a74756a017dbe2add34dbb (127.14.103.62:37179), c42e0c000a9b4b53a34d41a257df494d (127.14.103.60:40443)
I20260812 02:32:20.750663 23025 leader_election.cc:288] T 00000000000000000000000000000000 P c42e0c000a9b4b53a34d41a257df494d [CANDIDATE]: Term 2 election: Requested vote from peers 295ba34a46a74756a017dbe2add34dbb (127.14.103.62:37179), 198952c9f5644bc5a514d3f6d8a6a871 (127.14.103.61:35785)
I20260812 02:32:20.751081 23027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:20.750774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:37179 (local address 127.0.0.1:47730)
0812 02:32:20.750908 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:20.751003 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.62:37179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260812 02:32:20.751329 22213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c42e0c000a9b4b53a34d41a257df494d" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "198952c9f5644bc5a514d3f6d8a6a871"
I20260812 02:32:20.751406 23029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:20.750772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:37179 (local address 127.0.0.1:47724)
0812 02:32:20.751253 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:20.751306 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.62:37179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":407}
I20260812 02:32:20.751323 22425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "198952c9f5644bc5a514d3f6d8a6a871" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "c42e0c000a9b4b53a34d41a257df494d"
I20260812 02:32:20.751528 22213 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 198952c9f5644bc5a514d3f6d8a6a871 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate c42e0c000a9b4b53a34d41a257df494d in current term 2: Already voted for candidate 198952c9f5644bc5a514d3f6d8a6a871 in this term.
I20260812 02:32:20.751559 22425 raft_consensus.cc:2379] T 00000000000000000000000000000000 P c42e0c000a9b4b53a34d41a257df494d [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 198952c9f5644bc5a514d3f6d8a6a871 in current term 2: Already voted for candidate c42e0c000a9b4b53a34d41a257df494d in this term.
I20260812 02:32:20.752069 23029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:20.751852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:37179 (local address 127.0.0.1:47736)
0812 02:32:20.751976 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:20.752020 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.62:37179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260812 02:32:20.752205 22396 leader_election.cc:334] T 00000000000000000000000000000000 P c42e0c000a9b4b53a34d41a257df494d [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 295ba34a46a74756a017dbe2add34dbb (127.14.103.62:37179): Network error: Client connection negotiation failed: client connection to 127.14.103.62:37179: connect: Connection refused (error 111)
I20260812 02:32:20.752315 22396 leader_election.cc:302] T 00000000000000000000000000000000 P c42e0c000a9b4b53a34d41a257df494d [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c42e0c000a9b4b53a34d41a257df494d; no voters: 198952c9f5644bc5a514d3f6d8a6a871, 295ba34a46a74756a017dbe2add34dbb
I20260812 02:32:20.752480 23025 raft_consensus.cc:2726] T 00000000000000000000000000000000 P c42e0c000a9b4b53a34d41a257df494d [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260812 02:32:20.752525 23027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:20.752307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:37179 (local address 127.0.0.1:47752)
0812 02:32:20.752415 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:20.752463 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.62:37179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260812 02:32:20.752648 22183 leader_election.cc:334] T 00000000000000000000000000000000 P 198952c9f5644bc5a514d3f6d8a6a871 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 295ba34a46a74756a017dbe2add34dbb (127.14.103.62:37179): Network error: Client connection negotiation failed: client connection to 127.14.103.62:37179: connect: Connection refused (error 111)
I20260812 02:32:20.752755 22183 leader_election.cc:302] T 00000000000000000000000000000000 P 198952c9f5644bc5a514d3f6d8a6a871 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 198952c9f5644bc5a514d3f6d8a6a871; no voters: 295ba34a46a74756a017dbe2add34dbb, c42e0c000a9b4b53a34d41a257df494d
I20260812 02:32:20.752892 23026 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 198952c9f5644bc5a514d3f6d8a6a871 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260812 02:32:21.223734 23035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:21.222891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:37179 (local address 127.14.103.3:44511)
0812 02:32:21.223522 (+ 631us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:21.223609 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.62:37179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":142,"threads_started":1}
I20260812 02:32:21.225257 23036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:21.224719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:37179 (local address 127.14.103.2:56237)
0812 02:32:21.225073 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:21.225142 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.62:37179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":100,"threads_started":1}
I20260812 02:32:21.225951 23037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:21.225296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:37179 (local address 127.14.103.1:52441)
0812 02:32:21.225791 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:21.225855 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.62:37179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":78,"threads_started":1}
I20260812 02:32:22.225525 23038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:22.225051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:37179 (local address 127.14.103.3:60275)
0812 02:32:22.225343 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:22.225411 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.62:37179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":95,"threads_started":1}
I20260812 02:32:22.227805 23039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:22.227312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:37179 (local address 127.14.103.1:42783)
0812 02:32:22.227625 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:22.227695 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.62:37179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":110,"threads_started":1}
I20260812 02:32:22.228350 23040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:22.226936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:37179 (local address 127.14.103.2:35797)
0812 02:32:22.228046 (+ 1110us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:22.228111 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.62:37179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1029,"thread_start_us":67,"threads_started":1}
I20260812 02:32:22.449316 23041 raft_consensus.cc:482] T 00000000000000000000000000000000 P c42e0c000a9b4b53a34d41a257df494d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 02:32:22.449486 23041 raft_consensus.cc:504] T 00000000000000000000000000000000 P c42e0c000a9b4b53a34d41a257df494d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "295ba34a46a74756a017dbe2add34dbb" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 37179 } } peers { permanent_uuid: "198952c9f5644bc5a514d3f6d8a6a871" member_type: VOTER last_known_addr { host: "127.14.103.61" port: 35785 } attrs { promote: false } } peers { permanent_uuid: "c42e0c000a9b4b53a34d41a257df494d" member_type: VOTER last_known_addr { host: "127.14.103.60" port: 40443 } attrs { promote: false } }
I20260812 02:32:22.449925 23041 leader_election.cc:288] T 00000000000000000000000000000000 P c42e0c000a9b4b53a34d41a257df494d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 295ba34a46a74756a017dbe2add34dbb (127.14.103.62:37179), 198952c9f5644bc5a514d3f6d8a6a871 (127.14.103.61:35785)
I20260812 02:32:22.450476 22213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c42e0c000a9b4b53a34d41a257df494d" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "198952c9f5644bc5a514d3f6d8a6a871" is_pre_election: true
I20260812 02:32:22.450628 22213 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 198952c9f5644bc5a514d3f6d8a6a871 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c42e0c000a9b4b53a34d41a257df494d in term 2.
I20260812 02:32:22.450651 23042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:22.450196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:37179 (local address 127.0.0.1:48006)
0812 02:32:22.450442 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:22.450515 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.62:37179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":86,"threads_started":1}
I20260812 02:32:22.451124 22397 leader_election.cc:302] T 00000000000000000000000000000000 P c42e0c000a9b4b53a34d41a257df494d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 198952c9f5644bc5a514d3f6d8a6a871, c42e0c000a9b4b53a34d41a257df494d; no voters:
I20260812 02:32:22.451342 23041 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c42e0c000a9b4b53a34d41a257df494d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260812 02:32:22.451431 23041 raft_consensus.cc:482] T 00000000000000000000000000000000 P c42e0c000a9b4b53a34d41a257df494d [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 02:32:22.451485 23041 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c42e0c000a9b4b53a34d41a257df494d [term 2 FOLLOWER]: Advancing to term 3
I20260812 02:32:22.451749 23042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:22.451507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:37179 (local address 127.0.0.1:48014)
0812 02:32:22.451627 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:22.451688 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.62:37179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260812 02:32:22.451858 22396 leader_election.cc:334] T 00000000000000000000000000000000 P c42e0c000a9b4b53a34d41a257df494d [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 295ba34a46a74756a017dbe2add34dbb (127.14.103.62:37179): Network error: Client connection negotiation failed: client connection to 127.14.103.62:37179: connect: Connection refused (error 111)
I20260812 02:32:22.452788 23041 raft_consensus.cc:504] T 00000000000000000000000000000000 P c42e0c000a9b4b53a34d41a257df494d [term 3 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "295ba34a46a74756a017dbe2add34dbb" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 37179 } } peers { permanent_uuid: "198952c9f5644bc5a514d3f6d8a6a871" member_type: VOTER last_known_addr { host: "127.14.103.61" port: 35785 } attrs { promote: false } } peers { permanent_uuid: "c42e0c000a9b4b53a34d41a257df494d" member_type: VOTER last_known_addr { host: "127.14.103.60" port: 40443 } attrs { promote: false } }
I20260812 02:32:22.453166 23041 leader_election.cc:288] T 00000000000000000000000000000000 P c42e0c000a9b4b53a34d41a257df494d [CANDIDATE]: Term 3 election: Requested vote from peers 295ba34a46a74756a017dbe2add34dbb (127.14.103.62:37179), 198952c9f5644bc5a514d3f6d8a6a871 (127.14.103.61:35785)
I20260812 02:32:22.453583 23042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:22.453355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:37179 (local address 127.0.0.1:48028)
0812 02:32:22.453470 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:22.453525 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.62:37179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260812 02:32:22.453547 22213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c42e0c000a9b4b53a34d41a257df494d" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "198952c9f5644bc5a514d3f6d8a6a871"
I20260812 02:32:22.453727 22213 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 198952c9f5644bc5a514d3f6d8a6a871 [term 2 FOLLOWER]: Advancing to term 3
I20260812 02:32:22.454525 23042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:22.454276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:37179 (local address 127.0.0.1:48032)
0812 02:32:22.454411 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:22.454466 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.62:37179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260812 02:32:22.454630 22396 leader_election.cc:334] T 00000000000000000000000000000000 P c42e0c000a9b4b53a34d41a257df494d [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 295ba34a46a74756a017dbe2add34dbb (127.14.103.62:37179): Network error: Client connection negotiation failed: client connection to 127.14.103.62:37179: connect: Connection refused (error 111)
I20260812 02:32:22.455013 22213 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 198952c9f5644bc5a514d3f6d8a6a871 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c42e0c000a9b4b53a34d41a257df494d in term 3.
I20260812 02:32:22.455411 22397 leader_election.cc:302] T 00000000000000000000000000000000 P c42e0c000a9b4b53a34d41a257df494d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 198952c9f5644bc5a514d3f6d8a6a871, c42e0c000a9b4b53a34d41a257df494d; no voters: 295ba34a46a74756a017dbe2add34dbb
I20260812 02:32:22.455622 23041 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c42e0c000a9b4b53a34d41a257df494d [term 3 FOLLOWER]: Leader election won for term 3
I20260812 02:32:22.455849 23041 raft_consensus.cc:686] T 00000000000000000000000000000000 P c42e0c000a9b4b53a34d41a257df494d [term 3 LEADER]: Becoming Leader. State: Replica: c42e0c000a9b4b53a34d41a257df494d, State: Running, Role: LEADER
I20260812 02:32:22.456158 23041 consensus_queue.cc:237] T 00000000000000000000000000000000 P c42e0c000a9b4b53a34d41a257df494d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "295ba34a46a74756a017dbe2add34dbb" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 37179 } } peers { permanent_uuid: "198952c9f5644bc5a514d3f6d8a6a871" member_type: VOTER last_known_addr { host: "127.14.103.61" port: 35785 } attrs { promote: false } } peers { permanent_uuid: "c42e0c000a9b4b53a34d41a257df494d" member_type: VOTER last_known_addr { host: "127.14.103.60" port: 40443 } attrs { promote: false } }
I20260812 02:32:22.457556 23044 sys_catalog.cc:455] T 00000000000000000000000000000000 P c42e0c000a9b4b53a34d41a257df494d [sys.catalog]: SysCatalogTable state changed. Reason: New leader c42e0c000a9b4b53a34d41a257df494d. Latest consensus state: current_term: 3 leader_uuid: "c42e0c000a9b4b53a34d41a257df494d" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "295ba34a46a74756a017dbe2add34dbb" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 37179 } } peers { permanent_uuid: "198952c9f5644bc5a514d3f6d8a6a871" member_type: VOTER last_known_addr { host: "127.14.103.61" port: 35785 } attrs { promote: false } } peers { permanent_uuid: "c42e0c000a9b4b53a34d41a257df494d" member_type: VOTER last_known_addr { host: "127.14.103.60" port: 40443 } attrs { promote: false } } }
I20260812 02:32:22.457669 23044 sys_catalog.cc:458] T 00000000000000000000000000000000 P c42e0c000a9b4b53a34d41a257df494d [sys.catalog]: This master's current role is: LEADER
I20260812 02:32:22.457990 23047 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 02:32:22.460736 23047 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 02:32:22.461419 23047 catalog_manager.cc:1259] Loaded cluster ID: 6e4d29f3e17c4d179f9de01bd0f0be77
I20260812 02:32:22.461481 23047 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 02:32:22.462044 23047 catalog_manager.cc:1504] Loading token signing keys...
I20260812 02:32:22.462489 23047 catalog_manager.cc:5879] T 00000000000000000000000000000000 P c42e0c000a9b4b53a34d41a257df494d: Loaded TSK: 0
I20260812 02:32:22.463299 23047 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 02:32:22.553213 22415 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48930:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 02:32:22.556903 22415 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 02:32:22.560464 23042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:22.560176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:37179 (local address 127.0.0.1:48056)
0812 02:32:22.560316 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:22.560385 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.62:37179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260812 02:32:22.560693 22213 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 198952c9f5644bc5a514d3f6d8a6a871 [term 3 FOLLOWER]: Refusing update from remote peer c42e0c000a9b4b53a34d41a257df494d: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260812 02:32:22.561350 23042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:22.561082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:37179 (local address 127.0.0.1:48066)
0812 02:32:22.561218 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:22.561283 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.62:37179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260812 02:32:22.561269 23041 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c42e0c000a9b4b53a34d41a257df494d [LEADER]: Connected to new peer: Peer: permanent_uuid: "198952c9f5644bc5a514d3f6d8a6a871" member_type: VOTER last_known_addr { host: "127.14.103.61" port: 35785 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260812 02:32:22.561733 22396 consensus_peers.cc:487] T 00000000000000000000000000000000 P c42e0c000a9b4b53a34d41a257df494d -> Peer 295ba34a46a74756a017dbe2add34dbb (127.14.103.62:37179): Couldn't send request to peer 295ba34a46a74756a017dbe2add34dbb. Status: Network error: Client connection negotiation failed: client connection to 127.14.103.62:37179: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260812 02:32:22.564002 23051 sys_catalog.cc:455] T 00000000000000000000000000000000 P 198952c9f5644bc5a514d3f6d8a6a871 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c42e0c000a9b4b53a34d41a257df494d. Latest consensus state: current_term: 3 leader_uuid: "c42e0c000a9b4b53a34d41a257df494d" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "295ba34a46a74756a017dbe2add34dbb" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 37179 } } peers { permanent_uuid: "198952c9f5644bc5a514d3f6d8a6a871" member_type: VOTER last_known_addr { host: "127.14.103.61" port: 35785 } attrs { promote: false } } peers { permanent_uuid: "c42e0c000a9b4b53a34d41a257df494d" member_type: VOTER last_known_addr { host: "127.14.103.60" port: 40443 } attrs { promote: false } } }
I20260812 02:32:22.564347 23051 sys_catalog.cc:458] T 00000000000000000000000000000000 P 198952c9f5644bc5a514d3f6d8a6a871 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 02:32:22.565673 23044 sys_catalog.cc:455] T 00000000000000000000000000000000 P c42e0c000a9b4b53a34d41a257df494d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "c42e0c000a9b4b53a34d41a257df494d" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "295ba34a46a74756a017dbe2add34dbb" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 37179 } } peers { permanent_uuid: "198952c9f5644bc5a514d3f6d8a6a871" member_type: VOTER last_known_addr { host: "127.14.103.61" port: 35785 } attrs { promote: false } } peers { permanent_uuid: "c42e0c000a9b4b53a34d41a257df494d" member_type: VOTER last_known_addr { host: "127.14.103.60" port: 40443 } attrs { promote: false } } }
I20260812 02:32:22.565820 23044 sys_catalog.cc:458] T 00000000000000000000000000000000 P c42e0c000a9b4b53a34d41a257df494d [sys.catalog]: This master's current role is: LEADER
I20260812 02:32:22.566147 23051 sys_catalog.cc:455] T 00000000000000000000000000000000 P 198952c9f5644bc5a514d3f6d8a6a871 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "c42e0c000a9b4b53a34d41a257df494d" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "295ba34a46a74756a017dbe2add34dbb" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 37179 } } peers { permanent_uuid: "198952c9f5644bc5a514d3f6d8a6a871" member_type: VOTER last_known_addr { host: "127.14.103.61" port: 35785 } attrs { promote: false } } peers { permanent_uuid: "c42e0c000a9b4b53a34d41a257df494d" member_type: VOTER last_known_addr { host: "127.14.103.60" port: 40443 } attrs { promote: false } } }
I20260812 02:32:22.566320 23051 sys_catalog.cc:458] T 00000000000000000000000000000000 P 198952c9f5644bc5a514d3f6d8a6a871 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 02:32:22.577982 23042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:22.573270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.2:35055 (local address 127.0.0.1:43598)
0812 02:32:22.573438 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:22.573457 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 02:32:22.573556 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:32:22.573990 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:32:22.573993 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:32:22.574009 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:32:22.574120 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:22.574126 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:22.575477 (+ 1351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:22.575482 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:22.576654 (+ 1172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:22.576664 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:22.577564 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:22.577568 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:22.577660 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:22.577686 (+ 26us) client_negotiation.cc:769] Sending connection context
0812 02:32:22.577758 (+ 72us) client_negotiation.cc:240] Negotiation successful
0812 02:32:22.577832 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30,"mutex_wait_us":26}
I20260812 02:32:22.578047 23057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:22.573427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43598 (local address 127.14.103.2:35055)
0812 02:32:22.573775 (+ 348us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:22.573779 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:22.573790 (+ 11us) server_negotiation.cc:407] Connection header received
0812 02:32:22.573840 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:22.573842 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:22.573879 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:22.573981 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:32:22.574252 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:22.575349 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:22.576787 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:22.577420 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:22.577534 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:22.577608 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0812 02:32:22.577797 (+ 189us) server_negotiation.cc:299] Negotiation successful
0812 02:32:22.577888 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":158,"threads_started":1}
I20260812 02:32:22.578495 23058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:22.573927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.1:46041 (local address 127.0.0.1:56070)
0812 02:32:22.574281 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:22.574295 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 02:32:22.574384 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:32:22.574834 (+ 450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:32:22.574837 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:32:22.574856 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:32:22.574942 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:22.574947 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:22.575948 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:22.575952 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:22.577000 (+ 1048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:22.577011 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:22.577967 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:22.577971 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:22.578139 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:22.578169 (+ 30us) client_negotiation.cc:769] Sending connection context
0812 02:32:22.578249 (+ 80us) client_negotiation.cc:240] Negotiation successful
0812 02:32:22.578330 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"mutex_wait_us":33,"thread_start_us":133,"threads_started":1}
I20260812 02:32:22.578862 23060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:22.574332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56070 (local address 127.14.103.1:46041)
0812 02:32:22.574573 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:22.574576 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:22.574587 (+ 11us) server_negotiation.cc:407] Connection header received
0812 02:32:22.574630 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:22.574632 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:22.574669 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:22.574795 (+ 126us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:32:22.575077 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:22.575828 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:22.577101 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:22.577718 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:22.577843 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:22.577904 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0812 02:32:22.578666 (+ 762us) server_negotiation.cc:299] Negotiation successful
0812 02:32:22.578761 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":76,"threads_started":1}
I20260812 02:32:22.578855 22707 tablet_service.cc:1467] Processing CreateTablet for tablet efe13206010e4d4caf3df73ce6c79ac7 (DEFAULT_TABLE table=test-table [id=c4b4c14e0ffe4be49a793d7575a7ee85]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:32:22.579216 22707 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet efe13206010e4d4caf3df73ce6c79ac7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:32:22.579435 22549 tablet_service.cc:1467] Processing CreateTablet for tablet efe13206010e4d4caf3df73ce6c79ac7 (DEFAULT_TABLE table=test-table [id=c4b4c14e0ffe4be49a793d7575a7ee85]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:32:22.579731 22549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet efe13206010e4d4caf3df73ce6c79ac7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:32:22.580719 23061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:22.574614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41314 (local address 127.14.103.3:36861)
0812 02:32:22.574871 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:22.574877 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:22.575168 (+ 291us) server_negotiation.cc:407] Connection header received
0812 02:32:22.575265 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:22.575268 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:22.575312 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:22.575400 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:32:22.576177 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:22.576911 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:22.578198 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:22.578848 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:22.579008 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:22.579071 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0812 02:32:22.580205 (+ 1134us) server_negotiation.cc:299] Negotiation successful
0812 02:32:22.580375 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":60,"threads_started":1}
I20260812 02:32:22.581081 23059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:22.574272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.3:36861 (local address 127.0.0.1:41314)
0812 02:32:22.575066 (+ 794us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:22.575081 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 02:32:22.575180 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:32:22.575422 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:32:22.575425 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:32:22.575442 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:32:22.575579 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:22.575585 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:22.577024 (+ 1439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:22.577027 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:22.578052 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:22.578062 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:22.579569 (+ 1507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:22.579573 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:22.579939 (+ 366us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:22.579965 (+ 26us) client_negotiation.cc:769] Sending connection context
0812 02:32:22.580047 (+ 82us) client_negotiation.cc:240] Negotiation successful
0812 02:32:22.580931 (+ 884us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":668,"mutex_wait_us":49,"thread_start_us":88,"threads_started":1}
I20260812 02:32:22.581964 22865 tablet_service.cc:1467] Processing CreateTablet for tablet efe13206010e4d4caf3df73ce6c79ac7 (DEFAULT_TABLE table=test-table [id=c4b4c14e0ffe4be49a793d7575a7ee85]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:32:22.582252 22865 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet efe13206010e4d4caf3df73ce6c79ac7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:32:22.582511 23062 tablet_bootstrap.cc:492] T efe13206010e4d4caf3df73ce6c79ac7 P 0294bdfc2c3e476fb7c39c969dee950b: Bootstrap starting.
I20260812 02:32:22.583554 23063 tablet_bootstrap.cc:492] T efe13206010e4d4caf3df73ce6c79ac7 P 88fa3b5bb4714fa691fcc87e8d327b8f: Bootstrap starting.
I20260812 02:32:22.584270 23062 tablet_bootstrap.cc:654] T efe13206010e4d4caf3df73ce6c79ac7 P 0294bdfc2c3e476fb7c39c969dee950b: Neither blocks nor log segments found. Creating new log.
I20260812 02:32:22.584693 23063 tablet_bootstrap.cc:654] T efe13206010e4d4caf3df73ce6c79ac7 P 88fa3b5bb4714fa691fcc87e8d327b8f: Neither blocks nor log segments found. Creating new log.
I20260812 02:32:22.585000 23065 tablet_bootstrap.cc:492] T efe13206010e4d4caf3df73ce6c79ac7 P 9abf9de06df64b46b5db9181d885a31a: Bootstrap starting.
I20260812 02:32:22.586042 23065 tablet_bootstrap.cc:654] T efe13206010e4d4caf3df73ce6c79ac7 P 9abf9de06df64b46b5db9181d885a31a: Neither blocks nor log segments found. Creating new log.
I20260812 02:32:22.586136 23063 tablet_bootstrap.cc:492] T efe13206010e4d4caf3df73ce6c79ac7 P 88fa3b5bb4714fa691fcc87e8d327b8f: No bootstrap required, opened a new log
I20260812 02:32:22.586328 23063 ts_tablet_manager.cc:1397] T efe13206010e4d4caf3df73ce6c79ac7 P 88fa3b5bb4714fa691fcc87e8d327b8f: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260812 02:32:22.586807 23063 raft_consensus.cc:348] T efe13206010e4d4caf3df73ce6c79ac7 P 88fa3b5bb4714fa691fcc87e8d327b8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0294bdfc2c3e476fb7c39c969dee950b" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 35055 } } peers { permanent_uuid: "88fa3b5bb4714fa691fcc87e8d327b8f" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 46041 } } peers { permanent_uuid: "9abf9de06df64b46b5db9181d885a31a" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 36861 } }
I20260812 02:32:22.586941 23063 raft_consensus.cc:374] T efe13206010e4d4caf3df73ce6c79ac7 P 88fa3b5bb4714fa691fcc87e8d327b8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:32:22.587142 23063 raft_consensus.cc:729] T efe13206010e4d4caf3df73ce6c79ac7 P 88fa3b5bb4714fa691fcc87e8d327b8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88fa3b5bb4714fa691fcc87e8d327b8f, State: Initialized, Role: FOLLOWER
I20260812 02:32:22.587337 23063 consensus_queue.cc:260] T efe13206010e4d4caf3df73ce6c79ac7 P 88fa3b5bb4714fa691fcc87e8d327b8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0294bdfc2c3e476fb7c39c969dee950b" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 35055 } } peers { permanent_uuid: "88fa3b5bb4714fa691fcc87e8d327b8f" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 46041 } } peers { permanent_uuid: "9abf9de06df64b46b5db9181d885a31a" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 36861 } }
I20260812 02:32:22.587716 23063 ts_tablet_manager.cc:1428] T efe13206010e4d4caf3df73ce6c79ac7 P 88fa3b5bb4714fa691fcc87e8d327b8f: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260812 02:32:22.588013 23065 tablet_bootstrap.cc:492] T efe13206010e4d4caf3df73ce6c79ac7 P 9abf9de06df64b46b5db9181d885a31a: No bootstrap required, opened a new log
I20260812 02:32:22.588096 23065 ts_tablet_manager.cc:1397] T efe13206010e4d4caf3df73ce6c79ac7 P 9abf9de06df64b46b5db9181d885a31a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260812 02:32:22.588718 23065 raft_consensus.cc:348] T efe13206010e4d4caf3df73ce6c79ac7 P 9abf9de06df64b46b5db9181d885a31a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0294bdfc2c3e476fb7c39c969dee950b" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 35055 } } peers { permanent_uuid: "88fa3b5bb4714fa691fcc87e8d327b8f" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 46041 } } peers { permanent_uuid: "9abf9de06df64b46b5db9181d885a31a" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 36861 } }
I20260812 02:32:22.588845 23065 raft_consensus.cc:374] T efe13206010e4d4caf3df73ce6c79ac7 P 9abf9de06df64b46b5db9181d885a31a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:32:22.588989 23065 raft_consensus.cc:729] T efe13206010e4d4caf3df73ce6c79ac7 P 9abf9de06df64b46b5db9181d885a31a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9abf9de06df64b46b5db9181d885a31a, State: Initialized, Role: FOLLOWER
I20260812 02:32:22.589178 23065 consensus_queue.cc:260] T efe13206010e4d4caf3df73ce6c79ac7 P 9abf9de06df64b46b5db9181d885a31a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0294bdfc2c3e476fb7c39c969dee950b" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 35055 } } peers { permanent_uuid: "88fa3b5bb4714fa691fcc87e8d327b8f" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 46041 } } peers { permanent_uuid: "9abf9de06df64b46b5db9181d885a31a" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 36861 } }
I20260812 02:32:22.589633 23065 ts_tablet_manager.cc:1428] T efe13206010e4d4caf3df73ce6c79ac7 P 9abf9de06df64b46b5db9181d885a31a: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260812 02:32:22.590711 23039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:22.590384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:37179 (local address 127.14.103.1:58473)
0812 02:32:22.590550 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:22.590618 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.62:37179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260812 02:32:22.592316 22943 heartbeater.cc:503] Master 127.14.103.60:40443 was elected leader, sending a full tablet report...
I20260812 02:32:22.592905 23038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:22.592521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:37179 (local address 127.14.103.3:58485)
0812 02:32:22.592744 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:22.592811 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.62:37179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134}
I20260812 02:32:22.593111 23062 tablet_bootstrap.cc:492] T efe13206010e4d4caf3df73ce6c79ac7 P 0294bdfc2c3e476fb7c39c969dee950b: No bootstrap required, opened a new log
I20260812 02:32:22.593276 23062 ts_tablet_manager.cc:1397] T efe13206010e4d4caf3df73ce6c79ac7 P 0294bdfc2c3e476fb7c39c969dee950b: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20260812 02:32:22.593832 23062 raft_consensus.cc:348] T efe13206010e4d4caf3df73ce6c79ac7 P 0294bdfc2c3e476fb7c39c969dee950b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0294bdfc2c3e476fb7c39c969dee950b" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 35055 } } peers { permanent_uuid: "88fa3b5bb4714fa691fcc87e8d327b8f" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 46041 } } peers { permanent_uuid: "9abf9de06df64b46b5db9181d885a31a" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 36861 } }
I20260812 02:32:22.593976 23062 raft_consensus.cc:374] T efe13206010e4d4caf3df73ce6c79ac7 P 0294bdfc2c3e476fb7c39c969dee950b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:32:22.594025 23062 raft_consensus.cc:729] T efe13206010e4d4caf3df73ce6c79ac7 P 0294bdfc2c3e476fb7c39c969dee950b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0294bdfc2c3e476fb7c39c969dee950b, State: Initialized, Role: FOLLOWER
I20260812 02:32:22.594193 23062 consensus_queue.cc:260] T efe13206010e4d4caf3df73ce6c79ac7 P 0294bdfc2c3e476fb7c39c969dee950b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0294bdfc2c3e476fb7c39c969dee950b" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 35055 } } peers { permanent_uuid: "88fa3b5bb4714fa691fcc87e8d327b8f" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 46041 } } peers { permanent_uuid: "9abf9de06df64b46b5db9181d885a31a" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 36861 } }
I20260812 02:32:22.594646 23038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:22.594386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:37179 (local address 127.14.103.3:49927)
0812 02:32:22.594502 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:22.594566 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.62:37179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260812 02:32:22.594702 23062 ts_tablet_manager.cc:1428] T efe13206010e4d4caf3df73ce6c79ac7 P 0294bdfc2c3e476fb7c39c969dee950b: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260812 02:32:22.595369 23040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:22.595072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:37179 (local address 127.14.103.2:48197)
0812 02:32:22.595215 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:22.595299 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.62:37179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260812 02:32:22.596411 22782 heartbeater.cc:503] Master 127.14.103.60:40443 was elected leader, sending a full tablet report...
I20260812 02:32:22.597139 23040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:22.596848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:37179 (local address 127.14.103.2:42913)
0812 02:32:22.596990 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:22.597048 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.62:37179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260812 02:32:22.680866 23070 raft_consensus.cc:482] T efe13206010e4d4caf3df73ce6c79ac7 P 9abf9de06df64b46b5db9181d885a31a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 02:32:22.681128 23070 raft_consensus.cc:504] T efe13206010e4d4caf3df73ce6c79ac7 P 9abf9de06df64b46b5db9181d885a31a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0294bdfc2c3e476fb7c39c969dee950b" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 35055 } } peers { permanent_uuid: "88fa3b5bb4714fa691fcc87e8d327b8f" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 46041 } } peers { permanent_uuid: "9abf9de06df64b46b5db9181d885a31a" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 36861 } }
I20260812 02:32:22.682721 23070 leader_election.cc:288] T efe13206010e4d4caf3df73ce6c79ac7 P 9abf9de06df64b46b5db9181d885a31a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0294bdfc2c3e476fb7c39c969dee950b (127.14.103.2:35055), 88fa3b5bb4714fa691fcc87e8d327b8f (127.14.103.1:46041)
I20260812 02:32:22.687603 23057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:22.683234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.3:41349 (local address 127.14.103.2:35055)
0812 02:32:22.683443 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:22.683449 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:22.683567 (+ 118us) server_negotiation.cc:407] Connection header received
0812 02:32:22.683747 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:22.683751 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:22.683804 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:22.683912 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:32:22.684335 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:22.684899 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:22.686291 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:22.686860 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:22.687094 (+ 234us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:22.687269 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0812 02:32:22.687373 (+ 104us) server_negotiation.cc:299] Negotiation successful
0812 02:32:22.687473 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
I20260812 02:32:22.687608 23060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:22.683426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.3:46527 (local address 127.14.103.1:46041)
0812 02:32:22.683589 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:22.683595 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:22.683970 (+ 375us) server_negotiation.cc:407] Connection header received
0812 02:32:22.684155 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:22.684158 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:22.684214 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:22.684313 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:32:22.684634 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:22.685187 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:22.686384 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:22.686857 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:22.687039 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:22.687106 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0812 02:32:22.687388 (+ 282us) server_negotiation.cc:299] Negotiation successful
0812 02:32:22.687472 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260812 02:32:22.687603 23038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:22.683229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.2:35055 (local address 127.14.103.3:41349)
0812 02:32:22.683439 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:22.683459 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 02:32:22.683612 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:32:22.683925 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:32:22.683928 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:32:22.683946 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:32:22.684111 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:22.684118 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:22.685219 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:22.685225 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:22.686145 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:22.686155 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:22.687106 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:22.687109 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:22.687232 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:22.687255 (+ 23us) client_negotiation.cc:769] Sending connection context
0812 02:32:22.687357 (+ 102us) client_negotiation.cc:240] Negotiation successful
0812 02:32:22.687441 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67,"spinlock_wait_cycles":12544}
I20260812 02:32:22.687631 23072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:22.683241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.1:46041 (local address 127.14.103.3:46527)
0812 02:32:22.683855 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:22.683869 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 02:32:22.684020 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:32:22.684322 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:32:22.684326 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:32:22.684345 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:32:22.684492 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:22.684499 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:22.685323 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:22.685327 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:22.686219 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:22.686228 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:22.687055 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:22.687059 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:22.687177 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:22.687211 (+ 34us) client_negotiation.cc:769] Sending connection context
0812 02:32:22.687312 (+ 101us) client_negotiation.cc:240] Negotiation successful
0812 02:32:22.687411 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":536,"mutex_wait_us":200,"spinlock_wait_cycles":896,"thread_start_us":186,"threads_started":1}
I20260812 02:32:22.688359 22728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efe13206010e4d4caf3df73ce6c79ac7" candidate_uuid: "9abf9de06df64b46b5db9181d885a31a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0294bdfc2c3e476fb7c39c969dee950b" is_pre_election: true
I20260812 02:32:22.688431 22569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efe13206010e4d4caf3df73ce6c79ac7" candidate_uuid: "9abf9de06df64b46b5db9181d885a31a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88fa3b5bb4714fa691fcc87e8d327b8f" is_pre_election: true
I20260812 02:32:22.688627 22569 raft_consensus.cc:2454] T efe13206010e4d4caf3df73ce6c79ac7 P 88fa3b5bb4714fa691fcc87e8d327b8f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9abf9de06df64b46b5db9181d885a31a in term 0.
I20260812 02:32:22.688652 22728 raft_consensus.cc:2454] T efe13206010e4d4caf3df73ce6c79ac7 P 0294bdfc2c3e476fb7c39c969dee950b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9abf9de06df64b46b5db9181d885a31a in term 0.
I20260812 02:32:22.689100 22811 leader_election.cc:302] T efe13206010e4d4caf3df73ce6c79ac7 P 9abf9de06df64b46b5db9181d885a31a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0294bdfc2c3e476fb7c39c969dee950b, 88fa3b5bb4714fa691fcc87e8d327b8f, 9abf9de06df64b46b5db9181d885a31a; no voters:
I20260812 02:32:22.689468 23070 raft_consensus.cc:2781] T efe13206010e4d4caf3df73ce6c79ac7 P 9abf9de06df64b46b5db9181d885a31a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 02:32:22.689533 23070 raft_consensus.cc:482] T efe13206010e4d4caf3df73ce6c79ac7 P 9abf9de06df64b46b5db9181d885a31a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 02:32:22.689563 23070 raft_consensus.cc:3037] T efe13206010e4d4caf3df73ce6c79ac7 P 9abf9de06df64b46b5db9181d885a31a [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:32:22.690775 23070 raft_consensus.cc:504] T efe13206010e4d4caf3df73ce6c79ac7 P 9abf9de06df64b46b5db9181d885a31a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0294bdfc2c3e476fb7c39c969dee950b" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 35055 } } peers { permanent_uuid: "88fa3b5bb4714fa691fcc87e8d327b8f" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 46041 } } peers { permanent_uuid: "9abf9de06df64b46b5db9181d885a31a" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 36861 } }
I20260812 02:32:22.691431 23070 leader_election.cc:288] T efe13206010e4d4caf3df73ce6c79ac7 P 9abf9de06df64b46b5db9181d885a31a [CANDIDATE]: Term 1 election: Requested vote from peers 0294bdfc2c3e476fb7c39c969dee950b (127.14.103.2:35055), 88fa3b5bb4714fa691fcc87e8d327b8f (127.14.103.1:46041)
I20260812 02:32:22.691973 22569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efe13206010e4d4caf3df73ce6c79ac7" candidate_uuid: "9abf9de06df64b46b5db9181d885a31a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88fa3b5bb4714fa691fcc87e8d327b8f"
I20260812 02:32:22.692082 22728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efe13206010e4d4caf3df73ce6c79ac7" candidate_uuid: "9abf9de06df64b46b5db9181d885a31a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0294bdfc2c3e476fb7c39c969dee950b"
I20260812 02:32:22.692107 22569 raft_consensus.cc:3037] T efe13206010e4d4caf3df73ce6c79ac7 P 88fa3b5bb4714fa691fcc87e8d327b8f [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:32:22.692227 22728 raft_consensus.cc:3037] T efe13206010e4d4caf3df73ce6c79ac7 P 0294bdfc2c3e476fb7c39c969dee950b [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:32:22.693234 22569 raft_consensus.cc:2454] T efe13206010e4d4caf3df73ce6c79ac7 P 88fa3b5bb4714fa691fcc87e8d327b8f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9abf9de06df64b46b5db9181d885a31a in term 1.
I20260812 02:32:22.693245 22728 raft_consensus.cc:2454] T efe13206010e4d4caf3df73ce6c79ac7 P 0294bdfc2c3e476fb7c39c969dee950b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9abf9de06df64b46b5db9181d885a31a in term 1.
I20260812 02:32:22.693692 22813 leader_election.cc:302] T efe13206010e4d4caf3df73ce6c79ac7 P 9abf9de06df64b46b5db9181d885a31a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88fa3b5bb4714fa691fcc87e8d327b8f, 9abf9de06df64b46b5db9181d885a31a; no voters:
I20260812 02:32:22.693964 23070 raft_consensus.cc:2781] T efe13206010e4d4caf3df73ce6c79ac7 P 9abf9de06df64b46b5db9181d885a31a [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:32:22.694281 23070 raft_consensus.cc:686] T efe13206010e4d4caf3df73ce6c79ac7 P 9abf9de06df64b46b5db9181d885a31a [term 1 LEADER]: Becoming Leader. State: Replica: 9abf9de06df64b46b5db9181d885a31a, State: Running, Role: LEADER
I20260812 02:32:22.694572 23070 consensus_queue.cc:237] T efe13206010e4d4caf3df73ce6c79ac7 P 9abf9de06df64b46b5db9181d885a31a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0294bdfc2c3e476fb7c39c969dee950b" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 35055 } } peers { permanent_uuid: "88fa3b5bb4714fa691fcc87e8d327b8f" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 46041 } } peers { permanent_uuid: "9abf9de06df64b46b5db9181d885a31a" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 36861 } }
I20260812 02:32:22.695505 23072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:22.695178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:37179 (local address 127.14.103.3:57513)
0812 02:32:22.695360 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:22.695426 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.62:37179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
I20260812 02:32:22.697851 22406 catalog_manager.cc:5495] T efe13206010e4d4caf3df73ce6c79ac7 P 9abf9de06df64b46b5db9181d885a31a reported cstate change: term changed from 0 to 1, leader changed from <none> to 9abf9de06df64b46b5db9181d885a31a (127.14.103.3). New cstate: current_term: 1 leader_uuid: "9abf9de06df64b46b5db9181d885a31a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0294bdfc2c3e476fb7c39c969dee950b" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 35055 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88fa3b5bb4714fa691fcc87e8d327b8f" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 46041 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9abf9de06df64b46b5db9181d885a31a" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 36861 } health_report { overall_health: HEALTHY } } }
I20260812 02:32:22.737592 23061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:22.733852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41330 (local address 127.14.103.3:36861)
0812 02:32:22.733997 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:22.734001 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:22.734018 (+ 17us) server_negotiation.cc:407] Connection header received
0812 02:32:22.734151 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:22.734154 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:22.734209 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:22.734330 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 02:32:22.734587 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:22.735158 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:22.736051 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:22.736276 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:22.736392 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:22.736686 (+ 294us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 02:32:22.737167 (+ 481us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 02:32:22.737300 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0812 02:32:22.737369 (+ 69us) server_negotiation.cc:299] Negotiation successful
0812 02:32:22.737431 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260812 02:32:22.740868 22569 raft_consensus.cc:1261] T efe13206010e4d4caf3df73ce6c79ac7 P 88fa3b5bb4714fa691fcc87e8d327b8f [term 1 FOLLOWER]: Refusing update from remote peer 9abf9de06df64b46b5db9181d885a31a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:32:22.741281 22728 raft_consensus.cc:1261] T efe13206010e4d4caf3df73ce6c79ac7 P 0294bdfc2c3e476fb7c39c969dee950b [term 1 FOLLOWER]: Refusing update from remote peer 9abf9de06df64b46b5db9181d885a31a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:32:22.741524 23073 consensus_queue.cc:1035] T efe13206010e4d4caf3df73ce6c79ac7 P 9abf9de06df64b46b5db9181d885a31a [LEADER]: Connected to new peer: Peer: permanent_uuid: "88fa3b5bb4714fa691fcc87e8d327b8f" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 46041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:32:22.741868 23070 consensus_queue.cc:1035] T efe13206010e4d4caf3df73ce6c79ac7 P 9abf9de06df64b46b5db9181d885a31a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0294bdfc2c3e476fb7c39c969dee950b" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 35055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:32:22.742867 22622 heartbeater.cc:503] Master 127.14.103.60:40443 was elected leader, sending a full tablet report...
I20260812 02:32:22.744491 23040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:22.743774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:37179 (local address 127.14.103.2:57739)
0812 02:32:22.744334 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:22.744404 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.62:37179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":480}
I20260812 02:32:22.745365 23039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:22.744968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:37179 (local address 127.14.103.1:57435)
0812 02:32:22.745204 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:22.745265 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.62:37179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155}
I20260812 02:32:22.747838 23072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:22.747009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:37179 (local address 127.14.103.3:59643)
0812 02:32:22.747661 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:22.747725 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.62:37179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":566}
I20260812 02:32:22.748693 23072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:22.748305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:37179 (local address 127.14.103.3:54123)
0812 02:32:22.748575 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:22.748626 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.62:37179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192}
I20260812 02:32:22.751605 23039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:22.750946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:37179 (local address 127.14.103.1:59997)
0812 02:32:22.751428 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:22.751511 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.62:37179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":310}
I20260812 02:32:22.753695 23040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:22.753386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:37179 (local address 127.14.103.2:35093)
0812 02:32:22.753508 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:22.753582 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.62:37179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260812 02:32:22.756886 22406 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48930:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 02:32:22.757079 22406 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48930:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 02:32:22.761919 22406 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c42e0c000a9b4b53a34d41a257df494d: Sending DeleteTablet for 3 replicas of tablet efe13206010e4d4caf3df73ce6c79ac7
I20260812 02:32:22.762895 22707 tablet_service.cc:1514] Processing DeleteTablet for tablet efe13206010e4d4caf3df73ce6c79ac7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:32:22 UTC) from {username='oryx', principal='oryx/127.14.103.60@KRBTEST.COM'} at 127.0.0.1:43598
I20260812 02:32:22.763270 22865 tablet_service.cc:1514] Processing DeleteTablet for tablet efe13206010e4d4caf3df73ce6c79ac7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:32:22 UTC) from {username='oryx', principal='oryx/127.14.103.60@KRBTEST.COM'} at 127.0.0.1:41314
I20260812 02:32:22.763278 22549 tablet_service.cc:1514] Processing DeleteTablet for tablet efe13206010e4d4caf3df73ce6c79ac7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:32:22 UTC) from {username='oryx', principal='oryx/127.14.103.60@KRBTEST.COM'} at 127.0.0.1:56070
I20260812 02:32:22.763830 23091 tablet_replica.cc:330] T efe13206010e4d4caf3df73ce6c79ac7 P 0294bdfc2c3e476fb7c39c969dee950b: stopping tablet replica
I20260812 02:32:22.763877 23092 tablet_replica.cc:330] T efe13206010e4d4caf3df73ce6c79ac7 P 9abf9de06df64b46b5db9181d885a31a: stopping tablet replica
I20260812 02:32:22.764034 23092 raft_consensus.cc:2229] T efe13206010e4d4caf3df73ce6c79ac7 P 9abf9de06df64b46b5db9181d885a31a [term 1 LEADER]: Raft consensus shutting down.
I20260812 02:32:22.764060 23091 raft_consensus.cc:2229] T efe13206010e4d4caf3df73ce6c79ac7 P 0294bdfc2c3e476fb7c39c969dee950b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 02:32:22.764333 23091 raft_consensus.cc:2258] T efe13206010e4d4caf3df73ce6c79ac7 P 0294bdfc2c3e476fb7c39c969dee950b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:32:22.764418 23092 raft_consensus.cc:2258] T efe13206010e4d4caf3df73ce6c79ac7 P 9abf9de06df64b46b5db9181d885a31a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:32:22.764554 14748 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbep4M4/build/debug/bin/kudu with pid 22368
I20260812 02:32:22.767262 23091 ts_tablet_manager.cc:1905] T efe13206010e4d4caf3df73ce6c79ac7 P 0294bdfc2c3e476fb7c39c969dee950b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:32:22.767879 23092 ts_tablet_manager.cc:1905] T efe13206010e4d4caf3df73ce6c79ac7 P 9abf9de06df64b46b5db9181d885a31a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:32:22.770217 23091 ts_tablet_manager.cc:1918] T efe13206010e4d4caf3df73ce6c79ac7 P 0294bdfc2c3e476fb7c39c969dee950b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 02:32:22.770336 23091 log.cc:1196] T efe13206010e4d4caf3df73ce6c79ac7 P 0294bdfc2c3e476fb7c39c969dee950b: Deleting WAL directory at /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/minicluster-data/ts-1/wal/wals/efe13206010e4d4caf3df73ce6c79ac7
I20260812 02:32:22.770609 23091 ts_tablet_manager.cc:1939] T efe13206010e4d4caf3df73ce6c79ac7 P 0294bdfc2c3e476fb7c39c969dee950b: Deleting consensus metadata
I20260812 02:32:22.770872 23092 ts_tablet_manager.cc:1918] T efe13206010e4d4caf3df73ce6c79ac7 P 9abf9de06df64b46b5db9181d885a31a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 02:32:22.770998 23092 log.cc:1196] T efe13206010e4d4caf3df73ce6c79ac7 P 9abf9de06df64b46b5db9181d885a31a: Deleting WAL directory at /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786501898713699-14748-0/minicluster-data/ts-2/wal/wals/efe13206010e4d4caf3df73ce6c79ac7
I20260812 02:32:22.771360 23092 ts_tablet_manager.cc:1939] T efe13206010e4d4caf3df73ce6c79ac7 P 9abf9de06df64b46b5db9181d885a31a: Deleting consensus metadata
I20260812 02:32:22.771576 22397 catalog_manager.cc:4841] TS 0294bdfc2c3e476fb7c39c969dee950b (127.14.103.2:35055): tablet efe13206010e4d4caf3df73ce6c79ac7 (table test-table [id=c4b4c14e0ffe4be49a793d7575a7ee85]) successfully deleted
I20260812 02:32:22.772140 22396 catalog_manager.cc:4841] TS 9abf9de06df64b46b5db9181d885a31a (127.14.103.3:36861): tablet efe13206010e4d4caf3df73ce6c79ac7 (table test-table [id=c4b4c14e0ffe4be49a793d7575a7ee85]) successfully deleted
I20260812 02:32:22.781387 14748 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbep4M4/build/debug/bin/kudu with pid 22631
I20260812 02:32:22.781668 23059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:22.781305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.1:46041 (local address 127.0.0.1:56080)
0812 02:32:22.781486 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:22.781559 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.1:46041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260812 02:32:22.784865 22400 catalog_manager.cc:4584] TS 88fa3b5bb4714fa691fcc87e8d327b8f (127.14.103.1:46041): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet efe13206010e4d4caf3df73ce6c79ac7: Network error: Client connection negotiation failed: client connection to 127.14.103.1:46041: connect: Connection refused (error 111)
I20260812 02:32:22.792898 14748 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbep4M4/build/debug/bin/kudu with pid 22792
I20260812 02:32:22.803363 14748 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbep4M4/build/debug/bin/kudu with pid 22043
I20260812 02:32:22.813452 14748 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbep4M4/build/debug/bin/kudu with pid 22161
2026-08-12T02:32:22Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (8015 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501898713699-14748-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 02:32:22 dist-test-slave-2w3w krb5kdc[23096](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 02:32:22 dist-test-slave-2w3w krb5kdc[23096](info): set up 2 sockets
Aug 12 02:32:22 dist-test-slave-2w3w krb5kdc[23096](info): commencing operation
krb5kdc: starting...
W20260812 02:32:23.982005 14748 mini_kdc.cc:121] Time spent starting KDC: real 1.139s user 0.001s sys 0.009s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501898713699-14748-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501898713699-14748-0/krb5kdc/test-user.keytab.
Aug 12 02:32:24 dist-test-slave-2w3w krb5kdc[23096](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501944, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T02:32:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T02:32:24Z Disabled control of system clock
WARNING: no policy specified for oryx/127.14.103.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.103.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.103.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501898713699-14748-0/krb5kdc/oryx_127.14.103.62.keytab.
Entry for principal oryx/127.14.103.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501898713699-14748-0/krb5kdc/oryx_127.14.103.62.keytab.
I20260812 02:32:24.147354 14748 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbep4M4/build/debug/bin/kudu
/tmp/dist-test-taskbep4M4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501898713699-14748-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501898713699-14748-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501898713699-14748-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501898713699-14748-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.103.62:42935
--webserver_interface=127.14.103.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501898713699-14748-0/krb5kdc/oryx_127.14.103.62.keytab
--principal=oryx/127.14.103.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.103.20:39693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.103.62:42935
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501898713699-14748-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501898713699-14748-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501898713699-14748-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:32:24.278374 23111 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:32:24.278785 23111 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:32:24.278880 23111 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:32:24.284181 23111 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:32:24.284340 23111 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260812 02:32:24.284377 23111 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:32:24.284416 23111 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:32:24.284478 23111 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:32:24.284544 23111 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 02:32:24.291652 23111 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.103.20:39693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501898713699-14748-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501898713699-14748-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.103.62:42935
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501898713699-14748-0/krb5kdc/oryx_127.14.103.62.keytab
--principal=oryx/127.14.103.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.103.62:42935
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501898713699-14748-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.103.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23111
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501898713699-14748-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:32:24.293047 23111 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:32:24.294217 23111 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:32:24.303960 23120 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:32:24.304080 23116 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:32:24.304455 23111 server_base.cc:1034] running on GCE node
W20260812 02:32:24.304091 23117 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:32:24.305218 23111 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:32:24.306602 23111 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:32:24.307832 23111 hybrid_clock.cc:648] HybridClock initialized: now 1786501944307811 us; error 56 us; skew 500 ppm
Aug 12 02:32:24 dist-test-slave-2w3w krb5kdc[23096](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501944, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.103.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:32:24.311719 23111 init.cc:377] Logged in from keytab as oryx/127.14.103.62@KRBTEST.COM (short username oryx)
I20260812 02:32:24.312996 23111 webserver.cc:460] Webserver started at http://127.14.103.62:41623/ using document root <none> and password file <none>
I20260812 02:32:24.313699 23111 fs_manager.cc:359] Metadata directory not provided
I20260812 02:32:24.313815 23111 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:32:24.314096 23111 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:32:24.316581 23111 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501898713699-14748-0/minicluster-data/master-0/data/instance:
uuid: "ff23a1d724b647f786e885dc902c2d72"
format_stamp: "Formatted at 2026-08-12 02:32:24 on dist-test-slave-2w3w"
server_key: "99bfe40aef5b61d2c5fcd5bb358686ee"
server_key_iv: "2f7e74b16a4616cafba8c29e69354744"
server_key_version: "encryptionkey@0"
I20260812 02:32:24.317312 23111 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501898713699-14748-0/minicluster-data/master-0/wal/instance:
uuid: "ff23a1d724b647f786e885dc902c2d72"
format_stamp: "Formatted at 2026-08-12 02:32:24 on dist-test-slave-2w3w"
server_key: "99bfe40aef5b61d2c5fcd5bb358686ee"
server_key_iv: "2f7e74b16a4616cafba8c29e69354744"
server_key_version: "encryptionkey@0"
I20260812 02:32:24.321893 23111 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260812 02:32:24.325023 23128 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:32:24.326435 23111 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260812 02:32:24.326617 23111 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501898713699-14748-0/minicluster-data/master-0/data,/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501898713699-14748-0/minicluster-data/master-0/wal
uuid: "ff23a1d724b647f786e885dc902c2d72"
format_stamp: "Formatted at 2026-08-12 02:32:24 on dist-test-slave-2w3w"
server_key: "99bfe40aef5b61d2c5fcd5bb358686ee"
server_key_iv: "2f7e74b16a4616cafba8c29e69354744"
server_key_version: "encryptionkey@0"
I20260812 02:32:24.326762 23111 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501898713699-14748-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501898713699-14748-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501898713699-14748-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:32:24.340955 23111 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:32:24.345078 23111 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:32:24.345336 23111 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:32:24.355464 23111 rpc_server.cc:307] RPC server started. Bound to: 127.14.103.62:42935
I20260812 02:32:24.355552 23199 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.103.62:42935 every 8 connection(s)
I20260812 02:32:24.356769 23111 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501898713699-14748-0/minicluster-data/master-0/data/info.pb
I20260812 02:32:24.357169 14748 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbep4M4/build/debug/bin/kudu as pid 23111
I20260812 02:32:24.357316 14748 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501898713699-14748-0/minicluster-data/master-0/wal/instance
I20260812 02:32:24.357664 14748 external_mini_cluster.cc:1428] Setting key b395ce20c5714bf8efd6ff911facacc4
I20260812 02:32:24.360354 23200 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:32:24.367955 23200 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff23a1d724b647f786e885dc902c2d72: Bootstrap starting.
Aug 12 02:32:24 dist-test-slave-2w3w krb5kdc[23096](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501944, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.14.103.62@KRBTEST.COM
I20260812 02:32:24.371222 23200 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ff23a1d724b647f786e885dc902c2d72: Neither blocks nor log segments found. Creating new log.
I20260812 02:32:24.373054 23200 log.cc:824] T 00000000000000000000000000000000 P ff23a1d724b647f786e885dc902c2d72: Log is configured to *not* fsync() on all Append() calls
I20260812 02:32:24.376842 23200 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff23a1d724b647f786e885dc902c2d72: No bootstrap required, opened a new log
I20260812 02:32:24.377460 23203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:24.359268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50108 (local address 127.14.103.62:42935)
0812 02:32:24.359731 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:24.359748 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:24.359786 (+ 38us) server_negotiation.cc:407] Connection header received
0812 02:32:24.360473 (+ 687us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:24.360518 (+ 45us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:24.360901 (+ 383us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:24.361209 (+ 308us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:32:24.362339 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:24.364112 (+ 1773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:24.365378 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:24.366361 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:24.366517 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:24.369908 (+ 3391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:32:24.369917 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:32:24.369936 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:32:24.369981 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:32:24.372435 (+ 2454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:24.373181 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:24.373192 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:24.373202 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:24.373633 (+ 431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:24.374598 (+ 965us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:24.374610 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:24.374612 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:24.375257 (+ 645us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:32:24.375498 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0812 02:32:24.375853 (+ 355us) server_negotiation.cc:299] Negotiation successful
0812 02:32:24.376203 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":153,"threads_started":1}
I20260812 02:32:24.381610 23200 raft_consensus.cc:348] T 00000000000000000000000000000000 P ff23a1d724b647f786e885dc902c2d72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff23a1d724b647f786e885dc902c2d72" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 42935 } }
I20260812 02:32:24.381915 23200 raft_consensus.cc:374] T 00000000000000000000000000000000 P ff23a1d724b647f786e885dc902c2d72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:32:24.381955 23200 raft_consensus.cc:729] T 00000000000000000000000000000000 P ff23a1d724b647f786e885dc902c2d72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff23a1d724b647f786e885dc902c2d72, State: Initialized, Role: FOLLOWER
I20260812 02:32:24.382445 23200 consensus_queue.cc:260] T 00000000000000000000000000000000 P ff23a1d724b647f786e885dc902c2d72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff23a1d724b647f786e885dc902c2d72" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 42935 } }
I20260812 02:32:24.382603 23200 raft_consensus.cc:388] T 00000000000000000000000000000000 P ff23a1d724b647f786e885dc902c2d72 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 02:32:24.382653 23200 raft_consensus.cc:482] T 00000000000000000000000000000000 P ff23a1d724b647f786e885dc902c2d72 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 02:32:24.382756 23200 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ff23a1d724b647f786e885dc902c2d72 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:32:24.383956 23200 raft_consensus.cc:504] T 00000000000000000000000000000000 P ff23a1d724b647f786e885dc902c2d72 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff23a1d724b647f786e885dc902c2d72" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 42935 } }
I20260812 02:32:24.384411 23200 leader_election.cc:302] T 00000000000000000000000000000000 P ff23a1d724b647f786e885dc902c2d72 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ff23a1d724b647f786e885dc902c2d72; no voters:
I20260812 02:32:24.384862 23200 leader_election.cc:288] T 00000000000000000000000000000000 P ff23a1d724b647f786e885dc902c2d72 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 02:32:24.385563 23205 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ff23a1d724b647f786e885dc902c2d72 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:32:24.386075 23205 raft_consensus.cc:686] T 00000000000000000000000000000000 P ff23a1d724b647f786e885dc902c2d72 [term 1 LEADER]: Becoming Leader. State: Replica: ff23a1d724b647f786e885dc902c2d72, State: Running, Role: LEADER
I20260812 02:32:24.386336 23200 sys_catalog.cc:564] T 00000000000000000000000000000000 P ff23a1d724b647f786e885dc902c2d72 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:32:24.386786 23205 consensus_queue.cc:237] T 00000000000000000000000000000000 P ff23a1d724b647f786e885dc902c2d72 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff23a1d724b647f786e885dc902c2d72" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 42935 } }
I20260812 02:32:24.389539 23206 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff23a1d724b647f786e885dc902c2d72 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ff23a1d724b647f786e885dc902c2d72" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff23a1d724b647f786e885dc902c2d72" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 42935 } } }
I20260812 02:32:24.389746 23207 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff23a1d724b647f786e885dc902c2d72 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ff23a1d724b647f786e885dc902c2d72. Latest consensus state: current_term: 1 leader_uuid: "ff23a1d724b647f786e885dc902c2d72" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff23a1d724b647f786e885dc902c2d72" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 42935 } } }
I20260812 02:32:24.389846 23206 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff23a1d724b647f786e885dc902c2d72 [sys.catalog]: This master's current role is: LEADER
I20260812 02:32:24.389871 23207 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff23a1d724b647f786e885dc902c2d72 [sys.catalog]: This master's current role is: LEADER
W20260812 02:32:24.393610 23224 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ff23a1d724b647f786e885dc902c2d72: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 02:32:24.393761 23224 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 02:32:24.393893 23225 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 02:32:24.395082 23225 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 02:32:24.402642 23225 catalog_manager.cc:1347] Generated new cluster ID: f1589a2248444341943e0fb7240fec5e
I20260812 02:32:24.402763 23225 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 02:32:24.418857 23225 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 02:32:24.419965 23225 catalog_manager.cc:1504] Loading token signing keys...
I20260812 02:32:24.428755 23225 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ff23a1d724b647f786e885dc902c2d72: Generated new TSK 0
I20260812 02:32:24.429770 23225 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.14.103.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.103.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.103.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501898713699-14748-0/krb5kdc/oryx_127.14.103.1.keytab.
Entry for principal oryx/127.14.103.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501898713699-14748-0/krb5kdc/oryx_127.14.103.1.keytab.
I20260812 02:32:24.480703 14748 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbep4M4/build/debug/bin/kudu
/tmp/dist-test-taskbep4M4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501898713699-14748-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501898713699-14748-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501898713699-14748-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501898713699-14748-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.103.1:0
--local_ip_for_outbound_sockets=127.14.103.1
--webserver_interface=127.14.103.1
--webserver_port=0
--tserver_master_addrs=127.14.103.62:42935
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501898713699-14748-0/krb5kdc/oryx_127.14.103.1.keytab
--principal=oryx/127.14.103.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.103.20:39693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501898713699-14748-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501898713699-14748-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501898713699-14748-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:32:24.629812 23233 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:32:24.630367 23233 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:32:24.630478 23233 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:32:24.637428 23233 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:32:24.637554 23233 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:32:24.637604 23233 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 02:32:24.637681 23233 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.103.1
I20260812 02:32:24.644896 23233 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.103.20:39693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501898713699-14748-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501898713699-14748-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501898713699-14748-0/krb5kdc/oryx_127.14.103.1.keytab
--principal=oryx/127.14.103.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.103.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501898713699-14748-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.103.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.14.103.62:42935
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23233
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.103.1
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501898713699-14748-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:32:24.646203 23233 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:32:24.647446 23233 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:32:24.659487 23239 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:32:24.659488 23240 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:32:24.659502 23242 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:32:24.661024 23233 server_base.cc:1034] running on GCE node
I20260812 02:32:24.661495 23233 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:32:24.662194 23233 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:32:24.663383 23233 hybrid_clock.cc:648] HybridClock initialized: now 1786501944663350 us; error 51 us; skew 500 ppm
Aug 12 02:32:24 dist-test-slave-2w3w krb5kdc[23096](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501944, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.103.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:32:24.667212 23233 init.cc:377] Logged in from keytab as oryx/127.14.103.1@KRBTEST.COM (short username oryx)
I20260812 02:32:24.668663 23233 webserver.cc:460] Webserver started at http://127.14.103.1:39479/ using document root <none> and password file <none>
I20260812 02:32:24.669353 23233 fs_manager.cc:359] Metadata directory not provided
I20260812 02:32:24.669435 23233 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:32:24.669690 23233 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:32:24.672281 23233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501898713699-14748-0/minicluster-data/ts-0/data/instance:
uuid: "7a91f9c30fb24bc2abf32b1cc6fac06f"
format_stamp: "Formatted at 2026-08-12 02:32:24 on dist-test-slave-2w3w"
server_key: "6cdfc487d95c2c0498602e8c57f6a613"
server_key_iv: "2bc12a7e842d48fb25de66d0cd77ab46"
server_key_version: "encryptionkey@0"
I20260812 02:32:24.673643 23233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501898713699-14748-0/minicluster-data/ts-0/wal/instance:
uuid: "7a91f9c30fb24bc2abf32b1cc6fac06f"
format_stamp: "Formatted at 2026-08-12 02:32:24 on dist-test-slave-2w3w"
server_key: "6cdfc487d95c2c0498602e8c57f6a613"
server_key_iv: "2bc12a7e842d48fb25de66d0cd77ab46"
server_key_version: "encryptionkey@0"
I20260812 02:32:24.679301 23233 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260812 02:32:24.682431 23250 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:32:24.684288 23233 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 02:32:24.684490 23233 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501898713699-14748-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501898713699-14748-0/minicluster-data/ts-0/wal
uuid: "7a91f9c30fb24bc2abf32b1cc6fac06f"
format_stamp: "Formatted at 2026-08-12 02:32:24 on dist-test-slave-2w3w"
server_key: "6cdfc487d95c2c0498602e8c57f6a613"
server_key_iv: "2bc12a7e842d48fb25de66d0cd77ab46"
server_key_version: "encryptionkey@0"
I20260812 02:32:24.684648 23233 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501898713699-14748-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501898713699-14748-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501898713699-14748-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:32:24.707451 23233 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:32:24.712502 23233 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:32:24.712795 23233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:32:24.714591 23233 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:32:24.714681 23233 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:32:24.714763 23233 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:32:24.714809 23233 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 12 02:32:24 dist-test-slave-2w3w krb5kdc[23096](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501944, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.103.1@KRBTEST.COM for oryx/127.14.103.62@KRBTEST.COM
I20260812 02:32:24.727506 23233 rpc_server.cc:307] RPC server started. Bound to: 127.14.103.1:44325
I20260812 02:32:24.727583 23380 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.103.1:44325 every 8 connection(s)
I20260812 02:32:24.728612 23233 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501898713699-14748-0/minicluster-data/ts-0/data/info.pb
I20260812 02:32:24.732128 14748 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbep4M4/build/debug/bin/kudu as pid 23233
I20260812 02:32:24.732277 14748 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501898713699-14748-0/minicluster-data/ts-0/wal/instance
I20260812 02:32:24.732556 23203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:24.715773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.1:50795 (local address 127.14.103.62:42935)
0812 02:32:24.715969 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:24.715975 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:24.717027 (+ 1052us) server_negotiation.cc:407] Connection header received
0812 02:32:24.718610 (+ 1583us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:24.718615 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:24.718690 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:24.718819 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:32:24.720505 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:24.721091 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:24.722477 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:24.722783 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:24.722932 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:24.728003 (+ 5071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:32:24.728016 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:32:24.728020 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:32:24.728060 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:32:24.729905 (+ 1845us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:24.730620 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:24.730624 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:24.730628 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:24.730718 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:24.731211 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:24.731215 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:24.731217 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:24.731440 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:32:24.731558 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0812 02:32:24.732232 (+ 674us) server_negotiation.cc:299] Negotiation successful
0812 02:32:24.732394 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260812 02:32:24.732648 14748 external_mini_cluster.cc:1428] Setting key 46f5eeadf376062eb24a04a67ddc8c39
I20260812 02:32:24.733705 23259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:24.716217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:42935 (local address 127.14.103.1:50795)
0812 02:32:24.716780 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:24.716849 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0812 02:32:24.718300 (+ 1451us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:32:24.719086 (+ 786us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:32:24.719094 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:32:24.719564 (+ 470us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:32:24.720301 (+ 737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:24.720318 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:24.721252 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:24.721261 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:24.722322 (+ 1061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:24.722335 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:24.723036 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:24.723042 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:24.723493 (+ 451us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:24.724840 (+ 1347us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:32:24.724862 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:32:24.727593 (+ 2731us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:32:24.730075 (+ 2482us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:32:24.730083 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:32:24.730100 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:32:24.730493 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:32:24.730818 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:32:24.730823 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:32:24.730824 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:32:24.731092 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:32:24.731557 (+ 465us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:32:24.731565 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:32:24.731888 (+ 323us) client_negotiation.cc:769] Sending connection context
0812 02:32:24.732132 (+ 244us) client_negotiation.cc:240] Negotiation successful
0812 02:32:24.732546 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":118,"threads_started":1}
I20260812 02:32:24.735625 23381 heartbeater.cc:347] Connected to a master server at 127.14.103.62:42935
I20260812 02:32:24.735832 23381 heartbeater.cc:464] Registering TS with master...
I20260812 02:32:24.736398 23381 heartbeater.cc:511] Master 127.14.103.62:42935 requested a full tablet report, sending...
I20260812 02:32:24.738582 23148 ts_manager.cc:194] Registered new tserver with Master: 7a91f9c30fb24bc2abf32b1cc6fac06f (127.14.103.1:44325)
I20260812 02:32:24.740280 23148 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.14.103.1@KRBTEST.COM'} at 127.14.103.1:50795
WARNING: no policy specified for oryx/127.14.103.2@KRBTEST.COM; defaulting to no policy
I20260812 02:32:24.747568 23203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:24.738652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.1:51921 (local address 127.14.103.62:42935)
0812 02:32:24.739521 (+ 869us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:24.739529 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:24.739560 (+ 31us) server_negotiation.cc:407] Connection header received
0812 02:32:24.739637 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:24.739643 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:24.739736 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:24.739898 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:32:24.740597 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:24.741298 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:24.742281 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:24.742548 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:24.742678 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:24.744067 (+ 1389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:32:24.744070 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:32:24.744074 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:32:24.744111 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:32:24.745701 (+ 1590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:24.746199 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:24.746203 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:24.746205 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:24.746294 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:24.746597 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:24.746600 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:24.746602 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:24.746804 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:32:24.746922 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0812 02:32:24.747251 (+ 329us) server_negotiation.cc:299] Negotiation successful
0812 02:32:24.747377 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":690}
I20260812 02:32:24.747574 23389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:24.738463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:42935 (local address 127.14.103.1:51921)
0812 02:32:24.738855 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:24.738873 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 02:32:24.739078 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:32:24.739953 (+ 875us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:32:24.739957 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:32:24.740265 (+ 308us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:32:24.740423 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:24.740432 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:24.741472 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:24.741475 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:24.742142 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:24.742152 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:24.742688 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:24.742692 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:24.742798 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:24.743472 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:32:24.743486 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:32:24.743883 (+ 397us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:32:24.745842 (+ 1959us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:32:24.745846 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:32:24.745848 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:32:24.746089 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:32:24.746395 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:32:24.746398 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:32:24.746400 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:32:24.746497 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:32:24.746910 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:32:24.746913 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:32:24.747064 (+ 151us) client_negotiation.cc:769] Sending connection context
0812 02:32:24.747180 (+ 116us) client_negotiation.cc:240] Negotiation successful
0812 02:32:24.747365 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":148,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.103.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.103.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501898713699-14748-0/krb5kdc/oryx_127.14.103.2.keytab.
Entry for principal oryx/127.14.103.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501898713699-14748-0/krb5kdc/oryx_127.14.103.2.keytab.
I20260812 02:32:24.776033 14748 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbep4M4/build/debug/bin/kudu
/tmp/dist-test-taskbep4M4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501898713699-14748-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501898713699-14748-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501898713699-14748-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501898713699-14748-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.103.2:0
--local_ip_for_outbound_sockets=127.14.103.2
--webserver_interface=127.14.103.2
--webserver_port=0
--tserver_master_addrs=127.14.103.62:42935
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501898713699-14748-0/krb5kdc/oryx_127.14.103.2.keytab
--principal=oryx/127.14.103.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.103.20:39693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501898713699-14748-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501898713699-14748-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501898713699-14748-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:32:24.925607 23391 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:32:24.925957 23391 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:32:24.926010 23391 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:32:24.931340 23391 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:32:24.931452 23391 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:32:24.931524 23391 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 02:32:24.931602 23391 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.103.2
I20260812 02:32:24.938031 23391 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.103.20:39693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501898713699-14748-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501898713699-14748-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501898713699-14748-0/krb5kdc/oryx_127.14.103.2.keytab
--principal=oryx/127.14.103.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.103.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501898713699-14748-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.103.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.14.103.62:42935
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23391
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.103.2
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501898713699-14748-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:32:24.939383 23391 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:32:24.940558 23391 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:32:24.953902 23399 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:32:24.953907 23396 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:32:24.953902 23397 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:32:24.954437 23391 server_base.cc:1034] running on GCE node
I20260812 02:32:24.954946 23391 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:32:24.955766 23391 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:32:24.957010 23391 hybrid_clock.cc:648] HybridClock initialized: now 1786501944956979 us; error 46 us; skew 500 ppm
Aug 12 02:32:24 dist-test-slave-2w3w krb5kdc[23096](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501944, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.103.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:32:24.961030 23391 init.cc:377] Logged in from keytab as oryx/127.14.103.2@KRBTEST.COM (short username oryx)
I20260812 02:32:24.962145 23391 webserver.cc:460] Webserver started at http://127.14.103.2:44425/ using document root <none> and password file <none>
I20260812 02:32:24.962816 23391 fs_manager.cc:359] Metadata directory not provided
I20260812 02:32:24.962903 23391 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:32:24.963532 23391 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:32:24.966109 23391 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501898713699-14748-0/minicluster-data/ts-1/data/instance:
uuid: "133abf073a614cf9bfc3c2b6fa986166"
format_stamp: "Formatted at 2026-08-12 02:32:24 on dist-test-slave-2w3w"
server_key: "042a8572e178f3cac3872e55d6026fd9"
server_key_iv: "17f0d28e2a1f2194577d70e670071686"
server_key_version: "encryptionkey@0"
I20260812 02:32:24.966825 23391 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501898713699-14748-0/minicluster-data/ts-1/wal/instance:
uuid: "133abf073a614cf9bfc3c2b6fa986166"
format_stamp: "Formatted at 2026-08-12 02:32:24 on dist-test-slave-2w3w"
server_key: "042a8572e178f3cac3872e55d6026fd9"
server_key_iv: "17f0d28e2a1f2194577d70e670071686"
server_key_version: "encryptionkey@0"
I20260812 02:32:24.972472 23391 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260812 02:32:24.976380 23406 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:32:24.978133 23391 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20260812 02:32:24.978331 23391 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501898713699-14748-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501898713699-14748-0/minicluster-data/ts-1/wal
uuid: "133abf073a614cf9bfc3c2b6fa986166"
format_stamp: "Formatted at 2026-08-12 02:32:24 on dist-test-slave-2w3w"
server_key: "042a8572e178f3cac3872e55d6026fd9"
server_key_iv: "17f0d28e2a1f2194577d70e670071686"
server_key_version: "encryptionkey@0"
I20260812 02:32:24.978488 23391 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501898713699-14748-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501898713699-14748-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501898713699-14748-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:32:24.996387 23391 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:32:25.001160 23391 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:32:25.002054 23391 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:32:25.007997 23391 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:32:25.008340 23391 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:32:25.008456 23391 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:32:25.008494 23391 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 12 02:32:25 dist-test-slave-2w3w krb5kdc[23096](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501944, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.103.2@KRBTEST.COM for oryx/127.14.103.62@KRBTEST.COM
I20260812 02:32:25.021579 23391 rpc_server.cc:307] RPC server started. Bound to: 127.14.103.2:33543
I20260812 02:32:25.021667 23539 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.103.2:33543 every 8 connection(s)
I20260812 02:32:25.023718 23391 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501898713699-14748-0/minicluster-data/ts-1/data/info.pb
I20260812 02:32:25.027686 23203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:25.008071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.2:49675 (local address 127.14.103.62:42935)
0812 02:32:25.008483 (+ 412us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:25.008489 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:25.009377 (+ 888us) server_negotiation.cc:407] Connection header received
0812 02:32:25.010443 (+ 1066us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:25.010450 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:25.010523 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:25.010643 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:32:25.012458 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:25.013334 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:25.015617 (+ 2283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:25.016016 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:25.016151 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:25.020699 (+ 4548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:32:25.020703 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:32:25.020706 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:32:25.020738 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:32:25.023519 (+ 2781us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:25.024891 (+ 1372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:25.024897 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:25.024898 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:25.025062 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:25.025715 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:25.025722 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:25.025725 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:25.026328 (+ 603us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:32:25.026493 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0812 02:32:25.027368 (+ 875us) server_negotiation.cc:299] Negotiation successful
0812 02:32:25.027545 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302}
I20260812 02:32:25.028268 14748 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbep4M4/build/debug/bin/kudu as pid 23391
I20260812 02:32:25.028412 14748 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501898713699-14748-0/minicluster-data/ts-1/wal/instance
I20260812 02:32:25.028725 14748 external_mini_cluster.cc:1428] Setting key 2e00af58cb52d9e0e9ad047ffc2845f3
I20260812 02:32:25.028738 23414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:25.008512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:42935 (local address 127.14.103.2:49675)
0812 02:32:25.009163 (+ 651us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:25.009225 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0812 02:32:25.010170 (+ 945us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:32:25.010867 (+ 697us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:32:25.010875 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:32:25.011467 (+ 592us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:32:25.012176 (+ 709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:25.012202 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:25.013534 (+ 1332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:25.013540 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:25.015391 (+ 1851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:25.015412 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:25.016392 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:25.016397 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:25.016682 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:25.017932 (+ 1250us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:32:25.017957 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:32:25.020381 (+ 2424us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:32:25.023933 (+ 3552us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:32:25.023941 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:32:25.024024 (+ 83us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:32:25.024704 (+ 680us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:32:25.025202 (+ 498us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:32:25.025208 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:32:25.025212 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:32:25.025409 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:32:25.026528 (+ 1119us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:32:25.026534 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:32:25.026856 (+ 322us) client_negotiation.cc:769] Sending connection context
0812 02:32:25.027310 (+ 454us) client_negotiation.cc:240] Negotiation successful
0812 02:32:25.027650 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":161,"threads_started":1}
I20260812 02:32:25.031443 23540 heartbeater.cc:347] Connected to a master server at 127.14.103.62:42935
I20260812 02:32:25.031674 23540 heartbeater.cc:464] Registering TS with master...
I20260812 02:32:25.032431 23540 heartbeater.cc:511] Master 127.14.103.62:42935 requested a full tablet report, sending...
I20260812 02:32:25.034067 23146 ts_manager.cc:194] Registered new tserver with Master: 133abf073a614cf9bfc3c2b6fa986166 (127.14.103.2:33543)
I20260812 02:32:25.034919 23146 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.14.103.2@KRBTEST.COM'} at 127.14.103.2:49675
WARNING: no policy specified for oryx/127.14.103.3@KRBTEST.COM; defaulting to no policy
I20260812 02:32:25.044788 23203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:25.034882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.2:42805 (local address 127.14.103.62:42935)
0812 02:32:25.035047 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:25.035052 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:25.035676 (+ 624us) server_negotiation.cc:407] Connection header received
0812 02:32:25.035943 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:25.035947 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:25.035997 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:25.036100 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:32:25.036955 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:25.037473 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:25.038482 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:25.038718 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:25.038862 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:25.040501 (+ 1639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:32:25.040504 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:32:25.040506 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:32:25.040534 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:32:25.042404 (+ 1870us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:25.043041 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:25.043045 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:25.043047 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:25.043160 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:25.043601 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:25.043604 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:25.043605 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:25.043878 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:32:25.043974 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0812 02:32:25.044394 (+ 420us) server_negotiation.cc:299] Negotiation successful
0812 02:32:25.044583 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260812 02:32:25.044881 23547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:25.034801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:42935 (local address 127.14.103.2:42805)
0812 02:32:25.035584 (+ 783us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:25.035601 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 02:32:25.035820 (+ 219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:32:25.036261 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:32:25.036266 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:32:25.036590 (+ 324us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:32:25.036777 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:25.036785 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:25.037618 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:25.037622 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:25.038348 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:25.038359 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:25.038884 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:25.038888 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:25.039018 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:25.039863 (+ 845us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:32:25.039878 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:32:25.040293 (+ 415us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:32:25.042592 (+ 2299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:32:25.042596 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:32:25.042599 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:32:25.042885 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:32:25.043324 (+ 439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:32:25.043327 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:32:25.043328 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:32:25.043462 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:32:25.044006 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:32:25.044008 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:32:25.044129 (+ 121us) client_negotiation.cc:769] Sending connection context
0812 02:32:25.044305 (+ 176us) client_negotiation.cc:240] Negotiation successful
0812 02:32:25.044580 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":688,"thread_start_us":158,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.103.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.103.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501898713699-14748-0/krb5kdc/oryx_127.14.103.3.keytab.
Entry for principal oryx/127.14.103.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501898713699-14748-0/krb5kdc/oryx_127.14.103.3.keytab.
I20260812 02:32:25.072399 14748 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbep4M4/build/debug/bin/kudu
/tmp/dist-test-taskbep4M4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501898713699-14748-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501898713699-14748-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501898713699-14748-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501898713699-14748-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.103.3:0
--local_ip_for_outbound_sockets=127.14.103.3
--webserver_interface=127.14.103.3
--webserver_port=0
--tserver_master_addrs=127.14.103.62:42935
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501898713699-14748-0/krb5kdc/oryx_127.14.103.3.keytab
--principal=oryx/127.14.103.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.103.20:39693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501898713699-14748-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501898713699-14748-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501898713699-14748-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:32:25.208981 23549 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:32:25.209375 23549 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:32:25.209467 23549 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:32:25.214304 23549 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:32:25.214462 23549 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:32:25.214550 23549 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 02:32:25.214646 23549 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.103.3
I20260812 02:32:25.220754 23549 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.103.20:39693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501898713699-14748-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501898713699-14748-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501898713699-14748-0/krb5kdc/oryx_127.14.103.3.keytab
--principal=oryx/127.14.103.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.103.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501898713699-14748-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.103.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.14.103.62:42935
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23549
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.103.3
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501898713699-14748-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:32:25.221997 23549 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:32:25.223160 23549 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:32:25.234266 23555 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:32:25.234258 23554 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:32:25.234422 23558 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:32:25.234652 23549 server_base.cc:1034] running on GCE node
I20260812 02:32:25.235229 23549 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:32:25.236038 23549 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:32:25.237244 23549 hybrid_clock.cc:648] HybridClock initialized: now 1786501945237225 us; error 36 us; skew 500 ppm
Aug 12 02:32:25 dist-test-slave-2w3w krb5kdc[23096](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501945, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.103.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:32:25.240487 23549 init.cc:377] Logged in from keytab as oryx/127.14.103.3@KRBTEST.COM (short username oryx)
I20260812 02:32:25.241740 23549 webserver.cc:460] Webserver started at http://127.14.103.3:41271/ using document root <none> and password file <none>
I20260812 02:32:25.242427 23549 fs_manager.cc:359] Metadata directory not provided
I20260812 02:32:25.242497 23549 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:32:25.242725 23549 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:32:25.245090 23549 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501898713699-14748-0/minicluster-data/ts-2/data/instance:
uuid: "a5521494881948f68156f1c53716012f"
format_stamp: "Formatted at 2026-08-12 02:32:25 on dist-test-slave-2w3w"
server_key: "45b3002a15846487ca8b58484684b081"
server_key_iv: "9afba5252ff133166150a3056af82934"
server_key_version: "encryptionkey@0"
I20260812 02:32:25.245733 23549 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501898713699-14748-0/minicluster-data/ts-2/wal/instance:
uuid: "a5521494881948f68156f1c53716012f"
format_stamp: "Formatted at 2026-08-12 02:32:25 on dist-test-slave-2w3w"
server_key: "45b3002a15846487ca8b58484684b081"
server_key_iv: "9afba5252ff133166150a3056af82934"
server_key_version: "encryptionkey@0"
I20260812 02:32:25.249965 23549 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260812 02:32:25.252909 23566 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:32:25.254271 23549 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260812 02:32:25.254443 23549 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501898713699-14748-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501898713699-14748-0/minicluster-data/ts-2/wal
uuid: "a5521494881948f68156f1c53716012f"
format_stamp: "Formatted at 2026-08-12 02:32:25 on dist-test-slave-2w3w"
server_key: "45b3002a15846487ca8b58484684b081"
server_key_iv: "9afba5252ff133166150a3056af82934"
server_key_version: "encryptionkey@0"
I20260812 02:32:25.254580 23549 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501898713699-14748-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501898713699-14748-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501898713699-14748-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:32:25.269712 23549 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:32:25.274056 23549 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:32:25.274315 23549 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:32:25.276180 23549 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:32:25.276266 23549 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:32:25.276326 23549 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:32:25.276353 23549 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 12 02:32:25 dist-test-slave-2w3w krb5kdc[23096](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501945, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.103.3@KRBTEST.COM for oryx/127.14.103.62@KRBTEST.COM
I20260812 02:32:25.288890 23549 rpc_server.cc:307] RPC server started. Bound to: 127.14.103.3:38427
I20260812 02:32:25.289438 23700 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.103.3:38427 every 8 connection(s)
I20260812 02:32:25.290166 23549 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501898713699-14748-0/minicluster-data/ts-2/data/info.pb
I20260812 02:32:25.292114 14748 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbep4M4/build/debug/bin/kudu as pid 23549
I20260812 02:32:25.292287 14748 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501898713699-14748-0/minicluster-data/ts-2/wal/instance
I20260812 02:32:25.292801 14748 external_mini_cluster.cc:1428] Setting key 6f992a003fae4eade0a172626cae9aab
I20260812 02:32:25.294064 23203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:25.276834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.3:37627 (local address 127.14.103.62:42935)
0812 02:32:25.276981 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:25.276985 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:25.278160 (+ 1175us) server_negotiation.cc:407] Connection header received
0812 02:32:25.279343 (+ 1183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:25.279347 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:25.279424 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:25.279490 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:32:25.281278 (+ 1788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:25.281962 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:25.283507 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:25.283743 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:25.283890 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:25.287735 (+ 3845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:32:25.287740 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:32:25.287743 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:32:25.287800 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:32:25.290067 (+ 2267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:25.290938 (+ 871us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:25.290942 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:25.290944 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:25.291179 (+ 235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:25.292654 (+ 1475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:25.292657 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:25.292659 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:25.292891 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:32:25.293040 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0812 02:32:25.293633 (+ 593us) server_negotiation.cc:299] Negotiation successful
0812 02:32:25.293815 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260812 02:32:25.295135 23577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:25.277414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:42935 (local address 127.14.103.3:37627)
0812 02:32:25.277965 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:25.278011 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0812 02:32:25.278917 (+ 906us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:32:25.279945 (+ 1028us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:32:25.279952 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:32:25.280374 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:32:25.281014 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:25.281032 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:25.282106 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:25.282109 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:25.283353 (+ 1244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:25.283365 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:25.283996 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:25.284005 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:25.284446 (+ 441us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:25.285344 (+ 898us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:32:25.285368 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:32:25.287437 (+ 2069us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:32:25.290252 (+ 2815us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:32:25.290259 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:32:25.290275 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:32:25.290784 (+ 509us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:32:25.291303 (+ 519us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:32:25.291306 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:32:25.291308 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:32:25.291504 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:32:25.293040 (+ 1536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:32:25.293048 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:32:25.293343 (+ 295us) client_negotiation.cc:769] Sending connection context
0812 02:32:25.293557 (+ 214us) client_negotiation.cc:240] Negotiation successful
0812 02:32:25.293947 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":133,"threads_started":1}
I20260812 02:32:25.296546 23701 heartbeater.cc:347] Connected to a master server at 127.14.103.62:42935
I20260812 02:32:25.296734 23701 heartbeater.cc:464] Registering TS with master...
I20260812 02:32:25.297288 23701 heartbeater.cc:511] Master 127.14.103.62:42935 requested a full tablet report, sending...
I20260812 02:32:25.298507 23146 ts_manager.cc:194] Registered new tserver with Master: a5521494881948f68156f1c53716012f (127.14.103.3:38427)
I20260812 02:32:25.299427 23146 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.14.103.3@KRBTEST.COM'} at 127.14.103.3:37627
I20260812 02:32:25.307969 14748 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 02:32:25.308386 23203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:25.299154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.3:48037 (local address 127.14.103.62:42935)
0812 02:32:25.299282 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:25.299286 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:25.300035 (+ 749us) server_negotiation.cc:407] Connection header received
0812 02:32:25.300264 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:25.300267 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:25.300309 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:25.300385 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:32:25.301010 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:25.301698 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:25.302620 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:25.302862 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:25.302954 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:25.304409 (+ 1455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:32:25.304412 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:32:25.304415 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:32:25.304438 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:32:25.305963 (+ 1525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:25.306532 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:25.306536 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:25.306537 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:25.306667 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:25.307075 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:25.307079 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:25.307082 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:25.307588 (+ 506us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:32:25.307735 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0812 02:32:25.308059 (+ 324us) server_negotiation.cc:299] Negotiation successful
0812 02:32:25.308191 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260812 02:32:25.308405 23707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:25.299039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:42935 (local address 127.14.103.3:48037)
0812 02:32:25.299923 (+ 884us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:25.299943 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 02:32:25.300066 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:32:25.300442 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:32:25.300445 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:32:25.300724 (+ 279us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:32:25.300880 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:25.300889 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:25.301870 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:25.301873 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:25.302491 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:25.302500 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:25.303051 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:25.303055 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:25.303137 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:25.303792 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:32:25.303806 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:32:25.304238 (+ 432us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:32:25.306091 (+ 1853us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:32:25.306105 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:32:25.306107 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:32:25.306354 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:32:25.306782 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:32:25.306786 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:32:25.306788 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:32:25.306900 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:32:25.307722 (+ 822us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:32:25.307726 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:32:25.307856 (+ 130us) client_negotiation.cc:769] Sending connection context
0812 02:32:25.307980 (+ 124us) client_negotiation.cc:240] Negotiation successful
0812 02:32:25.308182 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":741,"thread_start_us":147,"threads_started":1}
Aug 12 02:32:25 dist-test-slave-2w3w krb5kdc[23096](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501945, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 12 02:32:25 dist-test-slave-2w3w krb5kdc[23096](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.14.103.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 02:32:25 dist-test-slave-2w3w krb5kdc[23096](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.14.103.62@KRBTEST.COM, Server not found in Kerberos database
W20260812 02:32:25.340277 23203 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:32:25.331400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50110 (local address 127.14.103.62:42935)
0812 02:32:25.331537 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:25.331541 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:25.331562 (+ 21us) server_negotiation.cc:407] Connection header received
0812 02:32:25.331606 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:25.331608 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:25.331648 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:25.331765 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:32:25.332733 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:25.333421 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:25.334470 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:25.334703 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:25.334820 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:25.339946 (+ 5126us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50110: BlockingRecv error: recv got EOF from 127.0.0.1:50110 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Aug 12 02:32:25 dist-test-slave-2w3w krb5kdc[23096](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501945, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.14.103.62@KRBTEST.COM
I20260812 02:32:25.353615 23203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:25.342630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50122 (local address 127.14.103.62:42935)
0812 02:32:25.342785 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:25.342792 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:25.342811 (+ 19us) server_negotiation.cc:407] Connection header received
0812 02:32:25.342883 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:25.342885 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:25.343010 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:25.343130 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:32:25.343837 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:25.344379 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:25.345319 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:25.345569 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:25.345691 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:25.349214 (+ 3523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:32:25.349219 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:32:25.349222 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:32:25.349274 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:32:25.351583 (+ 2309us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:25.352201 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:25.352204 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:25.352206 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:25.352308 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:25.352620 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:25.352623 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:25.352625 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:25.352841 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:32:25.353006 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0812 02:32:25.353281 (+ 275us) server_negotiation.cc:299] Negotiation successful
0812 02:32:25.353425 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260812 02:32:25.357609 23146 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50122:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 02:32:25.362182 23146 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 02:32:25.381214 23725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:25.374254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.1:44325 (local address 127.0.0.1:60422)
0812 02:32:25.375798 (+ 1544us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:25.375810 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 02:32:25.375931 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:32:25.376545 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:32:25.376548 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:32:25.376579 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:32:25.376846 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:25.376853 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:25.377910 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:25.377914 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:25.379329 (+ 1415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:25.379341 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:25.380518 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:25.380522 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:25.380659 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:25.380724 (+ 65us) client_negotiation.cc:769] Sending connection context
0812 02:32:25.380847 (+ 123us) client_negotiation.cc:240] Negotiation successful
0812 02:32:25.381020 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1483,"mutex_wait_us":727,"spinlock_wait_cycles":7552,"thread_start_us":135,"threads_started":1}
I20260812 02:32:25.381373 23730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:25.375235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58418 (local address 127.14.103.3:38427)
0812 02:32:25.375832 (+ 597us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:25.375848 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:25.375875 (+ 27us) server_negotiation.cc:407] Connection header received
0812 02:32:25.375960 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:25.375968 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:25.376183 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:25.376362 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:32:25.376694 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:25.377925 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:25.379802 (+ 1877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:25.380641 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:25.380767 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:25.380942 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0812 02:32:25.381096 (+ 154us) server_negotiation.cc:299] Negotiation successful
0812 02:32:25.381203 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":493,"thread_start_us":148,"threads_started":1}
I20260812 02:32:25.381889 23727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:25.374591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60422 (local address 127.14.103.1:44325)
0812 02:32:25.375328 (+ 737us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:25.375336 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:25.375960 (+ 624us) server_negotiation.cc:407] Connection header received
0812 02:32:25.376049 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:25.376054 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:25.376249 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:25.376397 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:32:25.376998 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:25.377714 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:25.379485 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:25.380339 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:25.380470 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:25.380644 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0812 02:32:25.381644 (+ 1000us) server_negotiation.cc:299] Negotiation successful
0812 02:32:25.381749 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":631,"thread_start_us":117,"threads_started":1}
I20260812 02:32:25.382654 23728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:25.374780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56914 (local address 127.14.103.2:33543)
0812 02:32:25.375945 (+ 1165us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:25.375958 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:25.375980 (+ 22us) server_negotiation.cc:407] Connection header received
0812 02:32:25.376049 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:25.376054 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:25.376249 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:25.376397 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:32:25.376997 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:25.378115 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:25.379775 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:25.380605 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:25.380755 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:25.380915 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0812 02:32:25.382115 (+ 1200us) server_negotiation.cc:299] Negotiation successful
0812 02:32:25.382333 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1033,"thread_start_us":137,"threads_started":1}
I20260812 02:32:25.383226 23726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:25.374254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.3:38427 (local address 127.0.0.1:58418)
0812 02:32:25.375371 (+ 1117us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:25.375438 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0812 02:32:25.375576 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:32:25.376356 (+ 780us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:32:25.376362 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:32:25.376385 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:32:25.376563 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:25.376571 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:25.378083 (+ 1512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:25.378087 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:25.379627 (+ 1540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:25.379637 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:25.380783 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:25.380787 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:25.380943 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:25.380972 (+ 29us) client_negotiation.cc:769] Sending connection context
0812 02:32:25.381045 (+ 73us) client_negotiation.cc:240] Negotiation successful
0812 02:32:25.381130 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":907,"mutex_wait_us":76,"thread_start_us":121,"threads_started":1}
I20260812 02:32:25.383565 23729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:25.374308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.2:33543 (local address 127.0.0.1:56914)
0812 02:32:25.375463 (+ 1155us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:25.375477 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 02:32:25.375585 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:32:25.376545 (+ 960us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:32:25.376548 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:32:25.376581 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:32:25.376845 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:25.376853 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:25.378242 (+ 1389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:25.378245 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:25.379624 (+ 1379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:25.379637 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:25.380975 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:25.380979 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:25.381084 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:25.381103 (+ 19us) client_negotiation.cc:769] Sending connection context
0812 02:32:25.381165 (+ 62us) client_negotiation.cc:240] Negotiation successful
0812 02:32:25.381227 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1076,"mutex_wait_us":166,"spinlock_wait_cycles":5248,"thread_start_us":139,"threads_started":1}
I20260812 02:32:25.385272 23305 tablet_service.cc:1467] Processing CreateTablet for tablet 0c5e00d3cb16447789f72324065d6032 (DEFAULT_TABLE table=test-table [id=9aa6d7ba895146d28b9e4aaf7b07fd04]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:32:25.386639 23305 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c5e00d3cb16447789f72324065d6032. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:32:25.386336 23460 tablet_service.cc:1467] Processing CreateTablet for tablet 0c5e00d3cb16447789f72324065d6032 (DEFAULT_TABLE table=test-table [id=9aa6d7ba895146d28b9e4aaf7b07fd04]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:32:25.386020 23625 tablet_service.cc:1467] Processing CreateTablet for tablet 0c5e00d3cb16447789f72324065d6032 (DEFAULT_TABLE table=test-table [id=9aa6d7ba895146d28b9e4aaf7b07fd04]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:32:25.387552 23460 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c5e00d3cb16447789f72324065d6032. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:32:25.387770 23625 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c5e00d3cb16447789f72324065d6032. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:32:25.395591 23731 tablet_bootstrap.cc:492] T 0c5e00d3cb16447789f72324065d6032 P 7a91f9c30fb24bc2abf32b1cc6fac06f: Bootstrap starting.
I20260812 02:32:25.397274 23733 tablet_bootstrap.cc:492] T 0c5e00d3cb16447789f72324065d6032 P 133abf073a614cf9bfc3c2b6fa986166: Bootstrap starting.
I20260812 02:32:25.397401 23734 tablet_bootstrap.cc:492] T 0c5e00d3cb16447789f72324065d6032 P a5521494881948f68156f1c53716012f: Bootstrap starting.
I20260812 02:32:25.398943 23731 tablet_bootstrap.cc:654] T 0c5e00d3cb16447789f72324065d6032 P 7a91f9c30fb24bc2abf32b1cc6fac06f: Neither blocks nor log segments found. Creating new log.
I20260812 02:32:25.399506 23733 tablet_bootstrap.cc:654] T 0c5e00d3cb16447789f72324065d6032 P 133abf073a614cf9bfc3c2b6fa986166: Neither blocks nor log segments found. Creating new log.
I20260812 02:32:25.399786 23734 tablet_bootstrap.cc:654] T 0c5e00d3cb16447789f72324065d6032 P a5521494881948f68156f1c53716012f: Neither blocks nor log segments found. Creating new log.
I20260812 02:32:25.400200 23731 log.cc:824] T 0c5e00d3cb16447789f72324065d6032 P 7a91f9c30fb24bc2abf32b1cc6fac06f: Log is configured to *not* fsync() on all Append() calls
I20260812 02:32:25.400383 23733 log.cc:824] T 0c5e00d3cb16447789f72324065d6032 P 133abf073a614cf9bfc3c2b6fa986166: Log is configured to *not* fsync() on all Append() calls
I20260812 02:32:25.400610 23734 log.cc:824] T 0c5e00d3cb16447789f72324065d6032 P a5521494881948f68156f1c53716012f: Log is configured to *not* fsync() on all Append() calls
I20260812 02:32:25.402849 23731 tablet_bootstrap.cc:492] T 0c5e00d3cb16447789f72324065d6032 P 7a91f9c30fb24bc2abf32b1cc6fac06f: No bootstrap required, opened a new log
I20260812 02:32:25.402849 23734 tablet_bootstrap.cc:492] T 0c5e00d3cb16447789f72324065d6032 P a5521494881948f68156f1c53716012f: No bootstrap required, opened a new log
I20260812 02:32:25.403146 23731 ts_tablet_manager.cc:1397] T 0c5e00d3cb16447789f72324065d6032 P 7a91f9c30fb24bc2abf32b1cc6fac06f: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260812 02:32:25.403146 23734 ts_tablet_manager.cc:1397] T 0c5e00d3cb16447789f72324065d6032 P a5521494881948f68156f1c53716012f: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260812 02:32:25.405804 23733 tablet_bootstrap.cc:492] T 0c5e00d3cb16447789f72324065d6032 P 133abf073a614cf9bfc3c2b6fa986166: No bootstrap required, opened a new log
I20260812 02:32:25.406334 23733 ts_tablet_manager.cc:1397] T 0c5e00d3cb16447789f72324065d6032 P 133abf073a614cf9bfc3c2b6fa986166: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260812 02:32:25.409705 23734 raft_consensus.cc:348] T 0c5e00d3cb16447789f72324065d6032 P a5521494881948f68156f1c53716012f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5521494881948f68156f1c53716012f" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 38427 } } peers { permanent_uuid: "7a91f9c30fb24bc2abf32b1cc6fac06f" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 44325 } } peers { permanent_uuid: "133abf073a614cf9bfc3c2b6fa986166" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 33543 } }
I20260812 02:32:25.410131 23734 raft_consensus.cc:374] T 0c5e00d3cb16447789f72324065d6032 P a5521494881948f68156f1c53716012f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:32:25.410199 23734 raft_consensus.cc:729] T 0c5e00d3cb16447789f72324065d6032 P a5521494881948f68156f1c53716012f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5521494881948f68156f1c53716012f, State: Initialized, Role: FOLLOWER
I20260812 02:32:25.410358 23731 raft_consensus.cc:348] T 0c5e00d3cb16447789f72324065d6032 P 7a91f9c30fb24bc2abf32b1cc6fac06f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5521494881948f68156f1c53716012f" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 38427 } } peers { permanent_uuid: "7a91f9c30fb24bc2abf32b1cc6fac06f" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 44325 } } peers { permanent_uuid: "133abf073a614cf9bfc3c2b6fa986166" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 33543 } }
I20260812 02:32:25.410622 23731 raft_consensus.cc:374] T 0c5e00d3cb16447789f72324065d6032 P 7a91f9c30fb24bc2abf32b1cc6fac06f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:32:25.410677 23731 raft_consensus.cc:729] T 0c5e00d3cb16447789f72324065d6032 P 7a91f9c30fb24bc2abf32b1cc6fac06f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a91f9c30fb24bc2abf32b1cc6fac06f, State: Initialized, Role: FOLLOWER
I20260812 02:32:25.410748 23734 consensus_queue.cc:260] T 0c5e00d3cb16447789f72324065d6032 P a5521494881948f68156f1c53716012f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5521494881948f68156f1c53716012f" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 38427 } } peers { permanent_uuid: "7a91f9c30fb24bc2abf32b1cc6fac06f" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 44325 } } peers { permanent_uuid: "133abf073a614cf9bfc3c2b6fa986166" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 33543 } }
I20260812 02:32:25.410811 23733 raft_consensus.cc:348] T 0c5e00d3cb16447789f72324065d6032 P 133abf073a614cf9bfc3c2b6fa986166 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5521494881948f68156f1c53716012f" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 38427 } } peers { permanent_uuid: "7a91f9c30fb24bc2abf32b1cc6fac06f" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 44325 } } peers { permanent_uuid: "133abf073a614cf9bfc3c2b6fa986166" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 33543 } }
I20260812 02:32:25.411154 23733 raft_consensus.cc:374] T 0c5e00d3cb16447789f72324065d6032 P 133abf073a614cf9bfc3c2b6fa986166 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:32:25.411114 23731 consensus_queue.cc:260] T 0c5e00d3cb16447789f72324065d6032 P 7a91f9c30fb24bc2abf32b1cc6fac06f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5521494881948f68156f1c53716012f" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 38427 } } peers { permanent_uuid: "7a91f9c30fb24bc2abf32b1cc6fac06f" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 44325 } } peers { permanent_uuid: "133abf073a614cf9bfc3c2b6fa986166" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 33543 } }
I20260812 02:32:25.411226 23733 raft_consensus.cc:729] T 0c5e00d3cb16447789f72324065d6032 P 133abf073a614cf9bfc3c2b6fa986166 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 133abf073a614cf9bfc3c2b6fa986166, State: Initialized, Role: FOLLOWER
I20260812 02:32:25.411648 23733 consensus_queue.cc:260] T 0c5e00d3cb16447789f72324065d6032 P 133abf073a614cf9bfc3c2b6fa986166 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5521494881948f68156f1c53716012f" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 38427 } } peers { permanent_uuid: "7a91f9c30fb24bc2abf32b1cc6fac06f" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 44325 } } peers { permanent_uuid: "133abf073a614cf9bfc3c2b6fa986166" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 33543 } }
I20260812 02:32:25.411738 23701 heartbeater.cc:503] Master 127.14.103.62:42935 was elected leader, sending a full tablet report...
I20260812 02:32:25.412053 23734 ts_tablet_manager.cc:1428] T 0c5e00d3cb16447789f72324065d6032 P a5521494881948f68156f1c53716012f: Time spent starting tablet: real 0.009s user 0.008s sys 0.000s
I20260812 02:32:25.412053 23731 ts_tablet_manager.cc:1428] T 0c5e00d3cb16447789f72324065d6032 P 7a91f9c30fb24bc2abf32b1cc6fac06f: Time spent starting tablet: real 0.009s user 0.008s sys 0.001s
I20260812 02:32:25.412407 23381 heartbeater.cc:503] Master 127.14.103.62:42935 was elected leader, sending a full tablet report...
I20260812 02:32:25.412695 23733 ts_tablet_manager.cc:1428] T 0c5e00d3cb16447789f72324065d6032 P 133abf073a614cf9bfc3c2b6fa986166: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260812 02:32:25.413033 23540 heartbeater.cc:503] Master 127.14.103.62:42935 was elected leader, sending a full tablet report...
I20260812 02:32:25.468538 23738 raft_consensus.cc:482] T 0c5e00d3cb16447789f72324065d6032 P a5521494881948f68156f1c53716012f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 02:32:25.468768 23738 raft_consensus.cc:504] T 0c5e00d3cb16447789f72324065d6032 P a5521494881948f68156f1c53716012f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5521494881948f68156f1c53716012f" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 38427 } } peers { permanent_uuid: "7a91f9c30fb24bc2abf32b1cc6fac06f" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 44325 } } peers { permanent_uuid: "133abf073a614cf9bfc3c2b6fa986166" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 33543 } }
I20260812 02:32:25.470036 23738 leader_election.cc:288] T 0c5e00d3cb16447789f72324065d6032 P a5521494881948f68156f1c53716012f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a91f9c30fb24bc2abf32b1cc6fac06f (127.14.103.1:44325), 133abf073a614cf9bfc3c2b6fa986166 (127.14.103.2:33543)
I20260812 02:32:25.474280 23577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:25.470379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.2:33543 (local address 127.14.103.3:54881)
0812 02:32:25.470527 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:25.470545 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 02:32:25.470663 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:32:25.471050 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:32:25.471053 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:32:25.471088 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:32:25.471371 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:25.471383 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:25.472221 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:25.472227 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:25.473063 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:25.473073 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:25.473784 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:25.473788 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:25.473907 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:25.473936 (+ 29us) client_negotiation.cc:769] Sending connection context
0812 02:32:25.474032 (+ 96us) client_negotiation.cc:240] Negotiation successful
0812 02:32:25.474118 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46,"spinlock_wait_cycles":7168}
I20260812 02:32:25.474280 23728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:25.470642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.3:54881 (local address 127.14.103.2:33543)
0812 02:32:25.470781 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:25.470794 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:25.470807 (+ 13us) server_negotiation.cc:407] Connection header received
0812 02:32:25.470857 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:25.470859 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:25.470897 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:25.471039 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:32:25.471540 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:25.472099 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:25.473215 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:25.473647 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:25.473798 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:25.473870 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0812 02:32:25.474051 (+ 181us) server_negotiation.cc:299] Negotiation successful
0812 02:32:25.474127 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260812 02:32:25.474665 23743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:25.470393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.1:44325 (local address 127.14.103.3:57385)
0812 02:32:25.470894 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:25.470905 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 02:32:25.471095 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:32:25.471375 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:32:25.471378 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:32:25.471394 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:32:25.471521 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:25.471526 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:25.472267 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:25.472270 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:25.473447 (+ 1177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:25.473464 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:25.474169 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:25.474180 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:25.474285 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:25.474309 (+ 24us) client_negotiation.cc:769] Sending connection context
0812 02:32:25.474407 (+ 98us) client_negotiation.cc:240] Negotiation successful
0812 02:32:25.474499 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":445,"mutex_wait_us":38,"spinlock_wait_cycles":3840,"thread_start_us":141,"threads_started":1}
I20260812 02:32:25.475087 23727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:25.470567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.3:57385 (local address 127.14.103.1:44325)
0812 02:32:25.470706 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:25.470711 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:25.471044 (+ 333us) server_negotiation.cc:407] Connection header received
0812 02:32:25.471212 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:25.471217 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:25.471266 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:25.471363 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:32:25.471634 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:25.472137 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:25.473593 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:25.474033 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:25.474177 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:25.474229 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0812 02:32:25.474770 (+ 541us) server_negotiation.cc:299] Negotiation successful
0812 02:32:25.474881 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260812 02:32:25.475193 23479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c5e00d3cb16447789f72324065d6032" candidate_uuid: "a5521494881948f68156f1c53716012f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "133abf073a614cf9bfc3c2b6fa986166" is_pre_election: true
I20260812 02:32:25.475569 23479 raft_consensus.cc:2454] T 0c5e00d3cb16447789f72324065d6032 P 133abf073a614cf9bfc3c2b6fa986166 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5521494881948f68156f1c53716012f in term 0.
I20260812 02:32:25.475874 23327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c5e00d3cb16447789f72324065d6032" candidate_uuid: "a5521494881948f68156f1c53716012f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a91f9c30fb24bc2abf32b1cc6fac06f" is_pre_election: true
I20260812 02:32:25.476193 23327 raft_consensus.cc:2454] T 0c5e00d3cb16447789f72324065d6032 P 7a91f9c30fb24bc2abf32b1cc6fac06f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5521494881948f68156f1c53716012f in term 0.
I20260812 02:32:25.476222 23569 leader_election.cc:302] T 0c5e00d3cb16447789f72324065d6032 P a5521494881948f68156f1c53716012f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 133abf073a614cf9bfc3c2b6fa986166, a5521494881948f68156f1c53716012f; no voters:
I20260812 02:32:25.476528 23738 raft_consensus.cc:2781] T 0c5e00d3cb16447789f72324065d6032 P a5521494881948f68156f1c53716012f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 02:32:25.476619 23738 raft_consensus.cc:482] T 0c5e00d3cb16447789f72324065d6032 P a5521494881948f68156f1c53716012f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 02:32:25.476686 23738 raft_consensus.cc:3037] T 0c5e00d3cb16447789f72324065d6032 P a5521494881948f68156f1c53716012f [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:32:25.478089 23738 raft_consensus.cc:504] T 0c5e00d3cb16447789f72324065d6032 P a5521494881948f68156f1c53716012f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5521494881948f68156f1c53716012f" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 38427 } } peers { permanent_uuid: "7a91f9c30fb24bc2abf32b1cc6fac06f" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 44325 } } peers { permanent_uuid: "133abf073a614cf9bfc3c2b6fa986166" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 33543 } }
I20260812 02:32:25.478569 23738 leader_election.cc:288] T 0c5e00d3cb16447789f72324065d6032 P a5521494881948f68156f1c53716012f [CANDIDATE]: Term 1 election: Requested vote from peers 7a91f9c30fb24bc2abf32b1cc6fac06f (127.14.103.1:44325), 133abf073a614cf9bfc3c2b6fa986166 (127.14.103.2:33543)
I20260812 02:32:25.479136 23479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c5e00d3cb16447789f72324065d6032" candidate_uuid: "a5521494881948f68156f1c53716012f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "133abf073a614cf9bfc3c2b6fa986166"
I20260812 02:32:25.479176 23327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c5e00d3cb16447789f72324065d6032" candidate_uuid: "a5521494881948f68156f1c53716012f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a91f9c30fb24bc2abf32b1cc6fac06f"
I20260812 02:32:25.479312 23327 raft_consensus.cc:3037] T 0c5e00d3cb16447789f72324065d6032 P 7a91f9c30fb24bc2abf32b1cc6fac06f [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:32:25.479311 23479 raft_consensus.cc:3037] T 0c5e00d3cb16447789f72324065d6032 P 133abf073a614cf9bfc3c2b6fa986166 [term 0 FOLLOWER]: Advancing to term 1
W20260812 02:32:25.480461 23383 tablet.cc:2369] T 0c5e00d3cb16447789f72324065d6032 P 7a91f9c30fb24bc2abf32b1cc6fac06f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 02:32:25.480818 23479 raft_consensus.cc:2454] T 0c5e00d3cb16447789f72324065d6032 P 133abf073a614cf9bfc3c2b6fa986166 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5521494881948f68156f1c53716012f in term 1.
I20260812 02:32:25.480818 23327 raft_consensus.cc:2454] T 0c5e00d3cb16447789f72324065d6032 P 7a91f9c30fb24bc2abf32b1cc6fac06f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5521494881948f68156f1c53716012f in term 1.
I20260812 02:32:25.481402 23568 leader_election.cc:302] T 0c5e00d3cb16447789f72324065d6032 P a5521494881948f68156f1c53716012f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a91f9c30fb24bc2abf32b1cc6fac06f, a5521494881948f68156f1c53716012f; no voters:
I20260812 02:32:25.481637 23738 raft_consensus.cc:2781] T 0c5e00d3cb16447789f72324065d6032 P a5521494881948f68156f1c53716012f [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:32:25.481920 23738 raft_consensus.cc:686] T 0c5e00d3cb16447789f72324065d6032 P a5521494881948f68156f1c53716012f [term 1 LEADER]: Becoming Leader. State: Replica: a5521494881948f68156f1c53716012f, State: Running, Role: LEADER
I20260812 02:32:25.482401 23738 consensus_queue.cc:237] T 0c5e00d3cb16447789f72324065d6032 P a5521494881948f68156f1c53716012f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5521494881948f68156f1c53716012f" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 38427 } } peers { permanent_uuid: "7a91f9c30fb24bc2abf32b1cc6fac06f" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 44325 } } peers { permanent_uuid: "133abf073a614cf9bfc3c2b6fa986166" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 33543 } }
I20260812 02:32:25.486104 23148 catalog_manager.cc:5495] T 0c5e00d3cb16447789f72324065d6032 P a5521494881948f68156f1c53716012f reported cstate change: term changed from 0 to 1, leader changed from <none> to a5521494881948f68156f1c53716012f (127.14.103.3). New cstate: current_term: 1 leader_uuid: "a5521494881948f68156f1c53716012f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5521494881948f68156f1c53716012f" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 38427 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a91f9c30fb24bc2abf32b1cc6fac06f" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 44325 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "133abf073a614cf9bfc3c2b6fa986166" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 33543 } health_report { overall_health: UNKNOWN } } }
I20260812 02:32:25.513221 23730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:25.509268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58428 (local address 127.14.103.3:38427)
0812 02:32:25.509451 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:25.509456 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:25.509472 (+ 16us) server_negotiation.cc:407] Connection header received
0812 02:32:25.509663 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:25.509666 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:25.509732 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:25.509889 (+ 157us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 02:32:25.510135 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:25.510658 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:25.511605 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:25.511848 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:25.511966 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:25.512263 (+ 297us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 02:32:25.512753 (+ 490us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 02:32:25.512899 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0812 02:32:25.512964 (+ 65us) server_negotiation.cc:299] Negotiation successful
0812 02:32:25.513030 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260812 02:32:25.520339 23327 raft_consensus.cc:1261] T 0c5e00d3cb16447789f72324065d6032 P 7a91f9c30fb24bc2abf32b1cc6fac06f [term 1 FOLLOWER]: Refusing update from remote peer a5521494881948f68156f1c53716012f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:32:25.520339 23479 raft_consensus.cc:1261] T 0c5e00d3cb16447789f72324065d6032 P 133abf073a614cf9bfc3c2b6fa986166 [term 1 FOLLOWER]: Refusing update from remote peer a5521494881948f68156f1c53716012f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:32:25.521296 23744 consensus_queue.cc:1035] T 0c5e00d3cb16447789f72324065d6032 P a5521494881948f68156f1c53716012f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a91f9c30fb24bc2abf32b1cc6fac06f" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 44325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:32:25.521570 23738 consensus_queue.cc:1035] T 0c5e00d3cb16447789f72324065d6032 P a5521494881948f68156f1c53716012f [LEADER]: Connected to new peer: Peer: permanent_uuid: "133abf073a614cf9bfc3c2b6fa986166" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 33543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260812 02:32:25.528697 23541 tablet.cc:2369] T 0c5e00d3cb16447789f72324065d6032 P 133abf073a614cf9bfc3c2b6fa986166: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 02:32:25.532553 23750 mvcc.cc:205] Tried to move back new op lower bound from 7317511968842850304 to 7317511968699400192. Current Snapshot: MvccSnapshot[applied={T|T < 7317511968842850304}]
I20260812 02:32:25.543948 23148 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50122:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 02:32:25.544131 23148 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50122:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 02:32:25.547338 23148 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ff23a1d724b647f786e885dc902c2d72: Sending DeleteTablet for 3 replicas of tablet 0c5e00d3cb16447789f72324065d6032
I20260812 02:32:25.548367 23460 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c5e00d3cb16447789f72324065d6032 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:32:25 UTC) from {username='oryx', principal='oryx/127.14.103.62@KRBTEST.COM'} at 127.0.0.1:56914
I20260812 02:32:25.548406 23305 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c5e00d3cb16447789f72324065d6032 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:32:25 UTC) from {username='oryx', principal='oryx/127.14.103.62@KRBTEST.COM'} at 127.0.0.1:60422
I20260812 02:32:25.548555 23625 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c5e00d3cb16447789f72324065d6032 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:32:25 UTC) from {username='oryx', principal='oryx/127.14.103.62@KRBTEST.COM'} at 127.0.0.1:58418
I20260812 02:32:25.548996 23760 tablet_replica.cc:330] T 0c5e00d3cb16447789f72324065d6032 P 7a91f9c30fb24bc2abf32b1cc6fac06f: stopping tablet replica
I20260812 02:32:25.549162 23761 tablet_replica.cc:330] T 0c5e00d3cb16447789f72324065d6032 P a5521494881948f68156f1c53716012f: stopping tablet replica
I20260812 02:32:25.549404 23760 raft_consensus.cc:2229] T 0c5e00d3cb16447789f72324065d6032 P 7a91f9c30fb24bc2abf32b1cc6fac06f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 02:32:25.549403 23761 raft_consensus.cc:2229] T 0c5e00d3cb16447789f72324065d6032 P a5521494881948f68156f1c53716012f [term 1 LEADER]: Raft consensus shutting down.
I20260812 02:32:25.549741 23760 raft_consensus.cc:2258] T 0c5e00d3cb16447789f72324065d6032 P 7a91f9c30fb24bc2abf32b1cc6fac06f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:32:25.549796 23761 raft_consensus.cc:2258] T 0c5e00d3cb16447789f72324065d6032 P a5521494881948f68156f1c53716012f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:32:25.550101 23148 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50122:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 02:32:25.550613 23148 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 02:32:25.551450 23760 ts_tablet_manager.cc:1905] T 0c5e00d3cb16447789f72324065d6032 P 7a91f9c30fb24bc2abf32b1cc6fac06f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:32:25.551450 23761 ts_tablet_manager.cc:1905] T 0c5e00d3cb16447789f72324065d6032 P a5521494881948f68156f1c53716012f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:32:25.556078 23760 ts_tablet_manager.cc:1918] T 0c5e00d3cb16447789f72324065d6032 P 7a91f9c30fb24bc2abf32b1cc6fac06f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 02:32:25.556084 23761 ts_tablet_manager.cc:1918] T 0c5e00d3cb16447789f72324065d6032 P a5521494881948f68156f1c53716012f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 02:32:25.556334 23760 log.cc:1196] T 0c5e00d3cb16447789f72324065d6032 P 7a91f9c30fb24bc2abf32b1cc6fac06f: Deleting WAL directory at /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501898713699-14748-0/minicluster-data/ts-0/wal/wals/0c5e00d3cb16447789f72324065d6032
I20260812 02:32:25.556332 23761 log.cc:1196] T 0c5e00d3cb16447789f72324065d6032 P a5521494881948f68156f1c53716012f: Deleting WAL directory at /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501898713699-14748-0/minicluster-data/ts-2/wal/wals/0c5e00d3cb16447789f72324065d6032
I20260812 02:32:25.557128 23760 ts_tablet_manager.cc:1939] T 0c5e00d3cb16447789f72324065d6032 P 7a91f9c30fb24bc2abf32b1cc6fac06f: Deleting consensus metadata
I20260812 02:32:25.557127 23761 ts_tablet_manager.cc:1939] T 0c5e00d3cb16447789f72324065d6032 P a5521494881948f68156f1c53716012f: Deleting consensus metadata
I20260812 02:32:25.559079 23759 tablet_replica.cc:330] T 0c5e00d3cb16447789f72324065d6032 P 133abf073a614cf9bfc3c2b6fa986166: stopping tablet replica
I20260812 02:32:25.559823 23130 catalog_manager.cc:4841] TS 7a91f9c30fb24bc2abf32b1cc6fac06f (127.14.103.1:44325): tablet 0c5e00d3cb16447789f72324065d6032 (table test-table [id=9aa6d7ba895146d28b9e4aaf7b07fd04]) successfully deleted
I20260812 02:32:25.560001 23759 raft_consensus.cc:2229] T 0c5e00d3cb16447789f72324065d6032 P 133abf073a614cf9bfc3c2b6fa986166 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 02:32:25.560408 23305 tablet_service.cc:1467] Processing CreateTablet for tablet d1d02e8e69804f5b96ed8ed55b45b15e (DEFAULT_TABLE table=test-table [id=a68f004ca989400cb32ad72b04549362]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:32:25.560694 23759 raft_consensus.cc:2258] T 0c5e00d3cb16447789f72324065d6032 P 133abf073a614cf9bfc3c2b6fa986166 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:32:25.560807 23305 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1d02e8e69804f5b96ed8ed55b45b15e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:32:25.561260 23129 catalog_manager.cc:4841] TS a5521494881948f68156f1c53716012f (127.14.103.3:38427): tablet 0c5e00d3cb16447789f72324065d6032 (table test-table [id=9aa6d7ba895146d28b9e4aaf7b07fd04]) successfully deleted
I20260812 02:32:25.562587 23460 tablet_service.cc:1467] Processing CreateTablet for tablet d1d02e8e69804f5b96ed8ed55b45b15e (DEFAULT_TABLE table=test-table [id=a68f004ca989400cb32ad72b04549362]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:32:25.562908 23460 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1d02e8e69804f5b96ed8ed55b45b15e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:32:25.563843 23731 tablet_bootstrap.cc:492] T d1d02e8e69804f5b96ed8ed55b45b15e P 7a91f9c30fb24bc2abf32b1cc6fac06f: Bootstrap starting.
I20260812 02:32:25.563772 23759 ts_tablet_manager.cc:1905] T 0c5e00d3cb16447789f72324065d6032 P 133abf073a614cf9bfc3c2b6fa986166: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:32:25.564986 23731 tablet_bootstrap.cc:654] T d1d02e8e69804f5b96ed8ed55b45b15e P 7a91f9c30fb24bc2abf32b1cc6fac06f: Neither blocks nor log segments found. Creating new log.
I20260812 02:32:25.565315 23625 tablet_service.cc:1467] Processing CreateTablet for tablet d1d02e8e69804f5b96ed8ed55b45b15e (DEFAULT_TABLE table=test-table [id=a68f004ca989400cb32ad72b04549362]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:32:25.565598 23733 tablet_bootstrap.cc:492] T d1d02e8e69804f5b96ed8ed55b45b15e P 133abf073a614cf9bfc3c2b6fa986166: Bootstrap starting.
I20260812 02:32:25.565623 23625 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1d02e8e69804f5b96ed8ed55b45b15e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:32:25.566622 23733 tablet_bootstrap.cc:654] T d1d02e8e69804f5b96ed8ed55b45b15e P 133abf073a614cf9bfc3c2b6fa986166: Neither blocks nor log segments found. Creating new log.
I20260812 02:32:25.568478 23759 ts_tablet_manager.cc:1918] T 0c5e00d3cb16447789f72324065d6032 P 133abf073a614cf9bfc3c2b6fa986166: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 02:32:25.568610 23759 log.cc:1196] T 0c5e00d3cb16447789f72324065d6032 P 133abf073a614cf9bfc3c2b6fa986166: Deleting WAL directory at /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501898713699-14748-0/minicluster-data/ts-1/wal/wals/0c5e00d3cb16447789f72324065d6032
I20260812 02:32:25.568953 23759 ts_tablet_manager.cc:1939] T 0c5e00d3cb16447789f72324065d6032 P 133abf073a614cf9bfc3c2b6fa986166: Deleting consensus metadata
I20260812 02:32:25.569561 23734 tablet_bootstrap.cc:492] T d1d02e8e69804f5b96ed8ed55b45b15e P a5521494881948f68156f1c53716012f: Bootstrap starting.
I20260812 02:32:25.569624 23731 tablet_bootstrap.cc:492] T d1d02e8e69804f5b96ed8ed55b45b15e P 7a91f9c30fb24bc2abf32b1cc6fac06f: No bootstrap required, opened a new log
I20260812 02:32:25.569731 23731 ts_tablet_manager.cc:1397] T d1d02e8e69804f5b96ed8ed55b45b15e P 7a91f9c30fb24bc2abf32b1cc6fac06f: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.001s
I20260812 02:32:25.570184 23731 raft_consensus.cc:348] T d1d02e8e69804f5b96ed8ed55b45b15e P 7a91f9c30fb24bc2abf32b1cc6fac06f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a91f9c30fb24bc2abf32b1cc6fac06f" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 44325 } } peers { permanent_uuid: "133abf073a614cf9bfc3c2b6fa986166" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 33543 } } peers { permanent_uuid: "a5521494881948f68156f1c53716012f" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 38427 } }
I20260812 02:32:25.570294 23731 raft_consensus.cc:374] T d1d02e8e69804f5b96ed8ed55b45b15e P 7a91f9c30fb24bc2abf32b1cc6fac06f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:32:25.570317 23731 raft_consensus.cc:729] T d1d02e8e69804f5b96ed8ed55b45b15e P 7a91f9c30fb24bc2abf32b1cc6fac06f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a91f9c30fb24bc2abf32b1cc6fac06f, State: Initialized, Role: FOLLOWER
I20260812 02:32:25.570463 23731 consensus_queue.cc:260] T d1d02e8e69804f5b96ed8ed55b45b15e P 7a91f9c30fb24bc2abf32b1cc6fac06f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a91f9c30fb24bc2abf32b1cc6fac06f" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 44325 } } peers { permanent_uuid: "133abf073a614cf9bfc3c2b6fa986166" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 33543 } } peers { permanent_uuid: "a5521494881948f68156f1c53716012f" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 38427 } }
I20260812 02:32:25.570705 23734 tablet_bootstrap.cc:654] T d1d02e8e69804f5b96ed8ed55b45b15e P a5521494881948f68156f1c53716012f: Neither blocks nor log segments found. Creating new log.
I20260812 02:32:25.570755 23731 ts_tablet_manager.cc:1428] T d1d02e8e69804f5b96ed8ed55b45b15e P 7a91f9c30fb24bc2abf32b1cc6fac06f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260812 02:32:25.572268 23734 tablet_bootstrap.cc:492] T d1d02e8e69804f5b96ed8ed55b45b15e P a5521494881948f68156f1c53716012f: No bootstrap required, opened a new log
I20260812 02:32:25.572364 23734 ts_tablet_manager.cc:1397] T d1d02e8e69804f5b96ed8ed55b45b15e P a5521494881948f68156f1c53716012f: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260812 02:32:25.572841 23734 raft_consensus.cc:348] T d1d02e8e69804f5b96ed8ed55b45b15e P a5521494881948f68156f1c53716012f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a91f9c30fb24bc2abf32b1cc6fac06f" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 44325 } } peers { permanent_uuid: "133abf073a614cf9bfc3c2b6fa986166" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 33543 } } peers { permanent_uuid: "a5521494881948f68156f1c53716012f" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 38427 } }
I20260812 02:32:25.572995 23734 raft_consensus.cc:374] T d1d02e8e69804f5b96ed8ed55b45b15e P a5521494881948f68156f1c53716012f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:32:25.573032 23734 raft_consensus.cc:729] T d1d02e8e69804f5b96ed8ed55b45b15e P a5521494881948f68156f1c53716012f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5521494881948f68156f1c53716012f, State: Initialized, Role: FOLLOWER
I20260812 02:32:25.574311 23131 catalog_manager.cc:4841] TS 133abf073a614cf9bfc3c2b6fa986166 (127.14.103.2:33543): tablet 0c5e00d3cb16447789f72324065d6032 (table test-table [id=9aa6d7ba895146d28b9e4aaf7b07fd04]) successfully deleted
I20260812 02:32:25.574679 23734 consensus_queue.cc:260] T d1d02e8e69804f5b96ed8ed55b45b15e P a5521494881948f68156f1c53716012f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a91f9c30fb24bc2abf32b1cc6fac06f" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 44325 } } peers { permanent_uuid: "133abf073a614cf9bfc3c2b6fa986166" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 33543 } } peers { permanent_uuid: "a5521494881948f68156f1c53716012f" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 38427 } }
I20260812 02:32:25.575649 23734 ts_tablet_manager.cc:1428] T d1d02e8e69804f5b96ed8ed55b45b15e P a5521494881948f68156f1c53716012f: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20260812 02:32:25.576495 23733 tablet_bootstrap.cc:492] T d1d02e8e69804f5b96ed8ed55b45b15e P 133abf073a614cf9bfc3c2b6fa986166: No bootstrap required, opened a new log
I20260812 02:32:25.576650 23733 ts_tablet_manager.cc:1397] T d1d02e8e69804f5b96ed8ed55b45b15e P 133abf073a614cf9bfc3c2b6fa986166: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.000s
I20260812 02:32:25.577158 23733 raft_consensus.cc:348] T d1d02e8e69804f5b96ed8ed55b45b15e P 133abf073a614cf9bfc3c2b6fa986166 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a91f9c30fb24bc2abf32b1cc6fac06f" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 44325 } } peers { permanent_uuid: "133abf073a614cf9bfc3c2b6fa986166" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 33543 } } peers { permanent_uuid: "a5521494881948f68156f1c53716012f" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 38427 } }
I20260812 02:32:25.577306 23733 raft_consensus.cc:374] T d1d02e8e69804f5b96ed8ed55b45b15e P 133abf073a614cf9bfc3c2b6fa986166 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:32:25.577343 23733 raft_consensus.cc:729] T d1d02e8e69804f5b96ed8ed55b45b15e P 133abf073a614cf9bfc3c2b6fa986166 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 133abf073a614cf9bfc3c2b6fa986166, State: Initialized, Role: FOLLOWER
I20260812 02:32:25.577498 23733 consensus_queue.cc:260] T d1d02e8e69804f5b96ed8ed55b45b15e P 133abf073a614cf9bfc3c2b6fa986166 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a91f9c30fb24bc2abf32b1cc6fac06f" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 44325 } } peers { permanent_uuid: "133abf073a614cf9bfc3c2b6fa986166" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 33543 } } peers { permanent_uuid: "a5521494881948f68156f1c53716012f" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 38427 } }
I20260812 02:32:25.577818 23733 ts_tablet_manager.cc:1428] T d1d02e8e69804f5b96ed8ed55b45b15e P 133abf073a614cf9bfc3c2b6fa986166: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260812 02:32:25.620630 23754 raft_consensus.cc:482] T d1d02e8e69804f5b96ed8ed55b45b15e P a5521494881948f68156f1c53716012f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 02:32:25.620884 23754 raft_consensus.cc:504] T d1d02e8e69804f5b96ed8ed55b45b15e P a5521494881948f68156f1c53716012f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a91f9c30fb24bc2abf32b1cc6fac06f" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 44325 } } peers { permanent_uuid: "133abf073a614cf9bfc3c2b6fa986166" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 33543 } } peers { permanent_uuid: "a5521494881948f68156f1c53716012f" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 38427 } }
I20260812 02:32:25.621493 23754 leader_election.cc:288] T d1d02e8e69804f5b96ed8ed55b45b15e P a5521494881948f68156f1c53716012f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a91f9c30fb24bc2abf32b1cc6fac06f (127.14.103.1:44325), 133abf073a614cf9bfc3c2b6fa986166 (127.14.103.2:33543)
I20260812 02:32:25.622027 23327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1d02e8e69804f5b96ed8ed55b45b15e" candidate_uuid: "a5521494881948f68156f1c53716012f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a91f9c30fb24bc2abf32b1cc6fac06f" is_pre_election: true
I20260812 02:32:25.622109 23479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1d02e8e69804f5b96ed8ed55b45b15e" candidate_uuid: "a5521494881948f68156f1c53716012f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "133abf073a614cf9bfc3c2b6fa986166" is_pre_election: true
I20260812 02:32:25.622212 23327 raft_consensus.cc:2454] T d1d02e8e69804f5b96ed8ed55b45b15e P 7a91f9c30fb24bc2abf32b1cc6fac06f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5521494881948f68156f1c53716012f in term 0.
I20260812 02:32:25.622316 23479 raft_consensus.cc:2454] T d1d02e8e69804f5b96ed8ed55b45b15e P 133abf073a614cf9bfc3c2b6fa986166 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5521494881948f68156f1c53716012f in term 0.
I20260812 02:32:25.622598 23568 leader_election.cc:302] T d1d02e8e69804f5b96ed8ed55b45b15e P a5521494881948f68156f1c53716012f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a91f9c30fb24bc2abf32b1cc6fac06f, a5521494881948f68156f1c53716012f; no voters:
I20260812 02:32:25.622816 23754 raft_consensus.cc:2781] T d1d02e8e69804f5b96ed8ed55b45b15e P a5521494881948f68156f1c53716012f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 02:32:25.622890 23754 raft_consensus.cc:482] T d1d02e8e69804f5b96ed8ed55b45b15e P a5521494881948f68156f1c53716012f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 02:32:25.622932 23754 raft_consensus.cc:3037] T d1d02e8e69804f5b96ed8ed55b45b15e P a5521494881948f68156f1c53716012f [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:32:25.624061 23754 raft_consensus.cc:504] T d1d02e8e69804f5b96ed8ed55b45b15e P a5521494881948f68156f1c53716012f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a91f9c30fb24bc2abf32b1cc6fac06f" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 44325 } } peers { permanent_uuid: "133abf073a614cf9bfc3c2b6fa986166" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 33543 } } peers { permanent_uuid: "a5521494881948f68156f1c53716012f" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 38427 } }
I20260812 02:32:25.624516 23754 leader_election.cc:288] T d1d02e8e69804f5b96ed8ed55b45b15e P a5521494881948f68156f1c53716012f [CANDIDATE]: Term 1 election: Requested vote from peers 7a91f9c30fb24bc2abf32b1cc6fac06f (127.14.103.1:44325), 133abf073a614cf9bfc3c2b6fa986166 (127.14.103.2:33543)
I20260812 02:32:25.624965 23479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1d02e8e69804f5b96ed8ed55b45b15e" candidate_uuid: "a5521494881948f68156f1c53716012f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "133abf073a614cf9bfc3c2b6fa986166"
I20260812 02:32:25.624950 23327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1d02e8e69804f5b96ed8ed55b45b15e" candidate_uuid: "a5521494881948f68156f1c53716012f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a91f9c30fb24bc2abf32b1cc6fac06f"
I20260812 02:32:25.625092 23327 raft_consensus.cc:3037] T d1d02e8e69804f5b96ed8ed55b45b15e P 7a91f9c30fb24bc2abf32b1cc6fac06f [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:32:25.625118 23479 raft_consensus.cc:3037] T d1d02e8e69804f5b96ed8ed55b45b15e P 133abf073a614cf9bfc3c2b6fa986166 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:32:25.626091 23479 raft_consensus.cc:2454] T d1d02e8e69804f5b96ed8ed55b45b15e P 133abf073a614cf9bfc3c2b6fa986166 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5521494881948f68156f1c53716012f in term 1.
I20260812 02:32:25.626334 23327 raft_consensus.cc:2454] T d1d02e8e69804f5b96ed8ed55b45b15e P 7a91f9c30fb24bc2abf32b1cc6fac06f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5521494881948f68156f1c53716012f in term 1.
I20260812 02:32:25.626513 23569 leader_election.cc:302] T d1d02e8e69804f5b96ed8ed55b45b15e P a5521494881948f68156f1c53716012f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 133abf073a614cf9bfc3c2b6fa986166, a5521494881948f68156f1c53716012f; no voters:
I20260812 02:32:25.626752 23754 raft_consensus.cc:2781] T d1d02e8e69804f5b96ed8ed55b45b15e P a5521494881948f68156f1c53716012f [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:32:25.626871 23754 raft_consensus.cc:686] T d1d02e8e69804f5b96ed8ed55b45b15e P a5521494881948f68156f1c53716012f [term 1 LEADER]: Becoming Leader. State: Replica: a5521494881948f68156f1c53716012f, State: Running, Role: LEADER
I20260812 02:32:25.627061 23754 consensus_queue.cc:237] T d1d02e8e69804f5b96ed8ed55b45b15e P a5521494881948f68156f1c53716012f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a91f9c30fb24bc2abf32b1cc6fac06f" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 44325 } } peers { permanent_uuid: "133abf073a614cf9bfc3c2b6fa986166" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 33543 } } peers { permanent_uuid: "a5521494881948f68156f1c53716012f" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 38427 } }
I20260812 02:32:25.629011 23146 catalog_manager.cc:5495] T d1d02e8e69804f5b96ed8ed55b45b15e P a5521494881948f68156f1c53716012f reported cstate change: term changed from 0 to 1, leader changed from <none> to a5521494881948f68156f1c53716012f (127.14.103.3). New cstate: current_term: 1 leader_uuid: "a5521494881948f68156f1c53716012f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a91f9c30fb24bc2abf32b1cc6fac06f" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 44325 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "133abf073a614cf9bfc3c2b6fa986166" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 33543 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5521494881948f68156f1c53716012f" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 38427 } health_report { overall_health: HEALTHY } } }
Aug 12 02:32:25 dist-test-slave-2w3w krb5kdc[23096](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501944, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.103.62@KRBTEST.COM for oryx/127.14.103.62@KRBTEST.COM
I20260812 02:32:25.674885 23203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:25.665087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50136 (local address 127.14.103.62:42935)
0812 02:32:25.665290 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:25.665296 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:25.665451 (+ 155us) server_negotiation.cc:407] Connection header received
0812 02:32:25.665630 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:25.665634 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:25.665683 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:25.665773 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:32:25.666505 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:25.667127 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:25.667959 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:25.668174 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:25.668310 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:25.670930 (+ 2620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:32:25.670934 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:32:25.670936 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:32:25.671001 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:32:25.672710 (+ 1709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:25.673402 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:25.673409 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:25.673411 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:25.673493 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:25.673830 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:25.673833 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:25.673834 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:25.674028 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:32:25.674139 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0812 02:32:25.674460 (+ 321us) server_negotiation.cc:299] Negotiation successful
0812 02:32:25.674703 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":126,"server-negotiator.queue_time_us":65}
I20260812 02:32:25.674896 23771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:25.664974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:42935 (local address 127.0.0.1:50136)
0812 02:32:25.665340 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:25.665355 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 02:32:25.665516 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:32:25.665799 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:32:25.665802 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:32:25.666263 (+ 461us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:32:25.666355 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:25.666372 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:25.667250 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:25.667254 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:25.667831 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:25.667840 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:25.668330 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:25.668333 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:25.668424 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:25.669103 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:32:25.669116 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:32:25.670737 (+ 1621us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:32:25.672958 (+ 2221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:32:25.672975 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:32:25.672982 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:32:25.673289 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:32:25.673589 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:32:25.673593 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:32:25.673595 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:32:25.673679 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:32:25.674152 (+ 473us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:32:25.674162 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:32:25.674284 (+ 122us) client_negotiation.cc:769] Sending connection context
0812 02:32:25.674413 (+ 129us) client_negotiation.cc:240] Negotiation successful
0812 02:32:25.674645 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"mutex_wait_us":4,"thread_start_us":152,"threads_started":1}
I20260812 02:32:25.680784 23146 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.14.103.62@KRBTEST.COM'} at 127.0.0.1:50136:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260812 02:32:25.681753 23146 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 02:32:25.687783 23305 tablet_service.cc:1467] Processing CreateTablet for tablet f003add4ac4749b4a61ff862953ca78e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=144fb8f2bba642c6b9df8c0e3aef9180]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260812 02:32:25.687985 23460 tablet_service.cc:1467] Processing CreateTablet for tablet f003add4ac4749b4a61ff862953ca78e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=144fb8f2bba642c6b9df8c0e3aef9180]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260812 02:32:25.687985 23625 tablet_service.cc:1467] Processing CreateTablet for tablet f003add4ac4749b4a61ff862953ca78e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=144fb8f2bba642c6b9df8c0e3aef9180]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260812 02:32:25.688289 23305 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f003add4ac4749b4a61ff862953ca78e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:32:25.688362 23625 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f003add4ac4749b4a61ff862953ca78e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:32:25.688362 23460 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f003add4ac4749b4a61ff862953ca78e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:32:25.691704 23734 tablet_bootstrap.cc:492] T f003add4ac4749b4a61ff862953ca78e P a5521494881948f68156f1c53716012f: Bootstrap starting.
I20260812 02:32:25.692296 23733 tablet_bootstrap.cc:492] T f003add4ac4749b4a61ff862953ca78e P 133abf073a614cf9bfc3c2b6fa986166: Bootstrap starting.
I20260812 02:32:25.693261 23734 tablet_bootstrap.cc:654] T f003add4ac4749b4a61ff862953ca78e P a5521494881948f68156f1c53716012f: Neither blocks nor log segments found. Creating new log.
I20260812 02:32:25.693387 23733 tablet_bootstrap.cc:654] T f003add4ac4749b4a61ff862953ca78e P 133abf073a614cf9bfc3c2b6fa986166: Neither blocks nor log segments found. Creating new log.
I20260812 02:32:25.694676 23733 tablet_bootstrap.cc:492] T f003add4ac4749b4a61ff862953ca78e P 133abf073a614cf9bfc3c2b6fa986166: No bootstrap required, opened a new log
I20260812 02:32:25.694694 23731 tablet_bootstrap.cc:492] T f003add4ac4749b4a61ff862953ca78e P 7a91f9c30fb24bc2abf32b1cc6fac06f: Bootstrap starting.
I20260812 02:32:25.694801 23733 ts_tablet_manager.cc:1397] T f003add4ac4749b4a61ff862953ca78e P 133abf073a614cf9bfc3c2b6fa986166: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260812 02:32:25.695310 23733 raft_consensus.cc:348] T f003add4ac4749b4a61ff862953ca78e P 133abf073a614cf9bfc3c2b6fa986166 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a91f9c30fb24bc2abf32b1cc6fac06f" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 44325 } } peers { permanent_uuid: "133abf073a614cf9bfc3c2b6fa986166" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 33543 } } peers { permanent_uuid: "a5521494881948f68156f1c53716012f" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 38427 } }
I20260812 02:32:25.695446 23733 raft_consensus.cc:374] T f003add4ac4749b4a61ff862953ca78e P 133abf073a614cf9bfc3c2b6fa986166 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:32:25.695478 23733 raft_consensus.cc:729] T f003add4ac4749b4a61ff862953ca78e P 133abf073a614cf9bfc3c2b6fa986166 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 133abf073a614cf9bfc3c2b6fa986166, State: Initialized, Role: FOLLOWER
I20260812 02:32:25.695653 23734 tablet_bootstrap.cc:492] T f003add4ac4749b4a61ff862953ca78e P a5521494881948f68156f1c53716012f: No bootstrap required, opened a new log
I20260812 02:32:25.695744 23734 ts_tablet_manager.cc:1397] T f003add4ac4749b4a61ff862953ca78e P a5521494881948f68156f1c53716012f: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260812 02:32:25.695858 23731 tablet_bootstrap.cc:654] T f003add4ac4749b4a61ff862953ca78e P 7a91f9c30fb24bc2abf32b1cc6fac06f: Neither blocks nor log segments found. Creating new log.
I20260812 02:32:25.696071 23733 consensus_queue.cc:260] T f003add4ac4749b4a61ff862953ca78e P 133abf073a614cf9bfc3c2b6fa986166 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a91f9c30fb24bc2abf32b1cc6fac06f" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 44325 } } peers { permanent_uuid: "133abf073a614cf9bfc3c2b6fa986166" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 33543 } } peers { permanent_uuid: "a5521494881948f68156f1c53716012f" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 38427 } }
I20260812 02:32:25.696220 23734 raft_consensus.cc:348] T f003add4ac4749b4a61ff862953ca78e P a5521494881948f68156f1c53716012f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a91f9c30fb24bc2abf32b1cc6fac06f" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 44325 } } peers { permanent_uuid: "133abf073a614cf9bfc3c2b6fa986166" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 33543 } } peers { permanent_uuid: "a5521494881948f68156f1c53716012f" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 38427 } }
I20260812 02:32:25.696341 23734 raft_consensus.cc:374] T f003add4ac4749b4a61ff862953ca78e P a5521494881948f68156f1c53716012f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:32:25.696400 23734 raft_consensus.cc:729] T f003add4ac4749b4a61ff862953ca78e P a5521494881948f68156f1c53716012f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5521494881948f68156f1c53716012f, State: Initialized, Role: FOLLOWER
I20260812 02:32:25.696355 23742 tablet_replica.cc:439] T f003add4ac4749b4a61ff862953ca78e P 133abf073a614cf9bfc3c2b6fa986166: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a91f9c30fb24bc2abf32b1cc6fac06f" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 44325 } } peers { permanent_uuid: "133abf073a614cf9bfc3c2b6fa986166" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 33543 } } peers { permanent_uuid: "a5521494881948f68156f1c53716012f" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 38427 } } }
I20260812 02:32:25.696514 23742 tablet_replica.cc:442] T f003add4ac4749b4a61ff862953ca78e P 133abf073a614cf9bfc3c2b6fa986166: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 02:32:25.696532 23734 consensus_queue.cc:260] T f003add4ac4749b4a61ff862953ca78e P a5521494881948f68156f1c53716012f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a91f9c30fb24bc2abf32b1cc6fac06f" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 44325 } } peers { permanent_uuid: "133abf073a614cf9bfc3c2b6fa986166" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 33543 } } peers { permanent_uuid: "a5521494881948f68156f1c53716012f" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 38427 } }
I20260812 02:32:25.696805 23734 ts_tablet_manager.cc:1428] T f003add4ac4749b4a61ff862953ca78e P a5521494881948f68156f1c53716012f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260812 02:32:25.697029 23754 tablet_replica.cc:439] T f003add4ac4749b4a61ff862953ca78e P a5521494881948f68156f1c53716012f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a91f9c30fb24bc2abf32b1cc6fac06f" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 44325 } } peers { permanent_uuid: "133abf073a614cf9bfc3c2b6fa986166" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 33543 } } peers { permanent_uuid: "a5521494881948f68156f1c53716012f" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 38427 } } }
I20260812 02:32:25.697168 23754 tablet_replica.cc:442] T f003add4ac4749b4a61ff862953ca78e P a5521494881948f68156f1c53716012f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 02:32:25.697218 23733 ts_tablet_manager.cc:1428] T f003add4ac4749b4a61ff862953ca78e P 133abf073a614cf9bfc3c2b6fa986166: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260812 02:32:25.700018 23731 tablet_bootstrap.cc:492] T f003add4ac4749b4a61ff862953ca78e P 7a91f9c30fb24bc2abf32b1cc6fac06f: No bootstrap required, opened a new log
I20260812 02:32:25.700137 23731 ts_tablet_manager.cc:1397] T f003add4ac4749b4a61ff862953ca78e P 7a91f9c30fb24bc2abf32b1cc6fac06f: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260812 02:32:25.700575 23731 raft_consensus.cc:348] T f003add4ac4749b4a61ff862953ca78e P 7a91f9c30fb24bc2abf32b1cc6fac06f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a91f9c30fb24bc2abf32b1cc6fac06f" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 44325 } } peers { permanent_uuid: "133abf073a614cf9bfc3c2b6fa986166" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 33543 } } peers { permanent_uuid: "a5521494881948f68156f1c53716012f" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 38427 } }
I20260812 02:32:25.700736 23731 raft_consensus.cc:374] T f003add4ac4749b4a61ff862953ca78e P 7a91f9c30fb24bc2abf32b1cc6fac06f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:32:25.700778 23731 raft_consensus.cc:729] T f003add4ac4749b4a61ff862953ca78e P 7a91f9c30fb24bc2abf32b1cc6fac06f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a91f9c30fb24bc2abf32b1cc6fac06f, State: Initialized, Role: FOLLOWER
I20260812 02:32:25.700920 23731 consensus_queue.cc:260] T f003add4ac4749b4a61ff862953ca78e P 7a91f9c30fb24bc2abf32b1cc6fac06f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a91f9c30fb24bc2abf32b1cc6fac06f" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 44325 } } peers { permanent_uuid: "133abf073a614cf9bfc3c2b6fa986166" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 33543 } } peers { permanent_uuid: "a5521494881948f68156f1c53716012f" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 38427 } }
I20260812 02:32:25.701301 23731 ts_tablet_manager.cc:1428] T f003add4ac4749b4a61ff862953ca78e P 7a91f9c30fb24bc2abf32b1cc6fac06f: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260812 02:32:25.701231 23739 tablet_replica.cc:439] T f003add4ac4749b4a61ff862953ca78e P 7a91f9c30fb24bc2abf32b1cc6fac06f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a91f9c30fb24bc2abf32b1cc6fac06f" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 44325 } } peers { permanent_uuid: "133abf073a614cf9bfc3c2b6fa986166" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 33543 } } peers { permanent_uuid: "a5521494881948f68156f1c53716012f" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 38427 } } }
I20260812 02:32:25.701393 23739 tablet_replica.cc:442] T f003add4ac4749b4a61ff862953ca78e P 7a91f9c30fb24bc2abf32b1cc6fac06f: This TxnStatusTablet replica's current role is: FOLLOWER
W20260812 02:32:25.792313 23702 tablet.cc:2369] T d1d02e8e69804f5b96ed8ed55b45b15e P a5521494881948f68156f1c53716012f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 02:32:25.931128 23739 raft_consensus.cc:482] T f003add4ac4749b4a61ff862953ca78e P 7a91f9c30fb24bc2abf32b1cc6fac06f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 02:32:25.931418 23739 raft_consensus.cc:504] T f003add4ac4749b4a61ff862953ca78e P 7a91f9c30fb24bc2abf32b1cc6fac06f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a91f9c30fb24bc2abf32b1cc6fac06f" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 44325 } } peers { permanent_uuid: "133abf073a614cf9bfc3c2b6fa986166" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 33543 } } peers { permanent_uuid: "a5521494881948f68156f1c53716012f" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 38427 } }
I20260812 02:32:25.932855 23739 leader_election.cc:288] T f003add4ac4749b4a61ff862953ca78e P 7a91f9c30fb24bc2abf32b1cc6fac06f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 133abf073a614cf9bfc3c2b6fa986166 (127.14.103.2:33543), a5521494881948f68156f1c53716012f (127.14.103.3:38427)
I20260812 02:32:25.933504 23754 raft_consensus.cc:482] T f003add4ac4749b4a61ff862953ca78e P a5521494881948f68156f1c53716012f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 02:32:25.933646 23754 raft_consensus.cc:504] T f003add4ac4749b4a61ff862953ca78e P a5521494881948f68156f1c53716012f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a91f9c30fb24bc2abf32b1cc6fac06f" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 44325 } } peers { permanent_uuid: "133abf073a614cf9bfc3c2b6fa986166" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 33543 } } peers { permanent_uuid: "a5521494881948f68156f1c53716012f" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 38427 } }
I20260812 02:32:25.934099 23754 leader_election.cc:288] T f003add4ac4749b4a61ff862953ca78e P a5521494881948f68156f1c53716012f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a91f9c30fb24bc2abf32b1cc6fac06f (127.14.103.1:44325), 133abf073a614cf9bfc3c2b6fa986166 (127.14.103.2:33543)
I20260812 02:32:25.934618 23479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f003add4ac4749b4a61ff862953ca78e" candidate_uuid: "a5521494881948f68156f1c53716012f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "133abf073a614cf9bfc3c2b6fa986166" is_pre_election: true
I20260812 02:32:25.934617 23327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f003add4ac4749b4a61ff862953ca78e" candidate_uuid: "a5521494881948f68156f1c53716012f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a91f9c30fb24bc2abf32b1cc6fac06f" is_pre_election: true
I20260812 02:32:25.934780 23327 raft_consensus.cc:2454] T f003add4ac4749b4a61ff862953ca78e P 7a91f9c30fb24bc2abf32b1cc6fac06f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5521494881948f68156f1c53716012f in term 0.
I20260812 02:32:25.934782 23479 raft_consensus.cc:2454] T f003add4ac4749b4a61ff862953ca78e P 133abf073a614cf9bfc3c2b6fa986166 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5521494881948f68156f1c53716012f in term 0.
I20260812 02:32:25.935317 23569 leader_election.cc:302] T f003add4ac4749b4a61ff862953ca78e P a5521494881948f68156f1c53716012f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 133abf073a614cf9bfc3c2b6fa986166, a5521494881948f68156f1c53716012f; no voters:
I20260812 02:32:25.935508 23754 raft_consensus.cc:2781] T f003add4ac4749b4a61ff862953ca78e P a5521494881948f68156f1c53716012f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 02:32:25.935560 23754 raft_consensus.cc:482] T f003add4ac4749b4a61ff862953ca78e P a5521494881948f68156f1c53716012f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 02:32:25.935593 23754 raft_consensus.cc:3037] T f003add4ac4749b4a61ff862953ca78e P a5521494881948f68156f1c53716012f [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:32:25.936909 23754 raft_consensus.cc:504] T f003add4ac4749b4a61ff862953ca78e P a5521494881948f68156f1c53716012f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a91f9c30fb24bc2abf32b1cc6fac06f" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 44325 } } peers { permanent_uuid: "133abf073a614cf9bfc3c2b6fa986166" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 33543 } } peers { permanent_uuid: "a5521494881948f68156f1c53716012f" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 38427 } }
I20260812 02:32:25.937347 23754 leader_election.cc:288] T f003add4ac4749b4a61ff862953ca78e P a5521494881948f68156f1c53716012f [CANDIDATE]: Term 1 election: Requested vote from peers 7a91f9c30fb24bc2abf32b1cc6fac06f (127.14.103.1:44325), 133abf073a614cf9bfc3c2b6fa986166 (127.14.103.2:33543)
I20260812 02:32:25.937774 23327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f003add4ac4749b4a61ff862953ca78e" candidate_uuid: "a5521494881948f68156f1c53716012f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a91f9c30fb24bc2abf32b1cc6fac06f"
I20260812 02:32:25.937920 23327 raft_consensus.cc:3037] T f003add4ac4749b4a61ff862953ca78e P 7a91f9c30fb24bc2abf32b1cc6fac06f [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:32:25.938515 23728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:25.933234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.1:44569 (local address 127.14.103.2:33543)
0812 02:32:25.933392 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:25.933396 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:25.933695 (+ 299us) server_negotiation.cc:407] Connection header received
0812 02:32:25.933951 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:25.933955 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:25.934014 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:25.934065 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:32:25.934990 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:25.935725 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:25.937254 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:25.937902 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:25.938032 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:25.938088 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0812 02:32:25.938298 (+ 210us) server_negotiation.cc:299] Negotiation successful
0812 02:32:25.938386 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260812 02:32:25.938562 23730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:25.933252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.1:33033 (local address 127.14.103.3:38427)
0812 02:32:25.933520 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:25.933525 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:25.933665 (+ 140us) server_negotiation.cc:407] Connection header received
0812 02:32:25.933890 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:25.933894 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:25.933947 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:25.933997 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:32:25.934672 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:25.935455 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:25.937203 (+ 1748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:25.937943 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:25.938077 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:25.938154 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0812 02:32:25.938375 (+ 221us) server_negotiation.cc:299] Negotiation successful
0812 02:32:25.938453 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180}
I20260812 02:32:25.938575 23775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:25.933129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.3:38427 (local address 127.14.103.1:33033)
0812 02:32:25.933555 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:25.933572 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 02:32:25.933754 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:32:25.934158 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:32:25.934163 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:32:25.934183 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:32:25.934356 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:25.934365 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:25.935621 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:25.935627 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:25.937064 (+ 1437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:25.937078 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:25.938141 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:25.938145 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:25.938232 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:25.938261 (+ 29us) client_negotiation.cc:769] Sending connection context
0812 02:32:25.938340 (+ 79us) client_negotiation.cc:240] Negotiation successful
0812 02:32:25.938418 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":337,"mutex_wait_us":3,"thread_start_us":135,"threads_started":1}
I20260812 02:32:25.939136 23776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:25.933130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.2:33543 (local address 127.14.103.1:44569)
0812 02:32:25.933571 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:25.933591 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 02:32:25.933819 (+ 228us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:32:25.934211 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:32:25.934216 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:32:25.934234 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:32:25.934539 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:25.934545 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:25.935875 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:25.935879 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:25.937157 (+ 1278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:25.937167 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:25.938044 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:25.938047 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:25.938147 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:25.938173 (+ 26us) client_negotiation.cc:769] Sending connection context
0812 02:32:25.938840 (+ 667us) client_negotiation.cc:240] Negotiation successful
0812 02:32:25.938933 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":372,"mutex_wait_us":228,"thread_start_us":141,"threads_started":1}
I20260812 02:32:25.939076 23479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f003add4ac4749b4a61ff862953ca78e" candidate_uuid: "a5521494881948f68156f1c53716012f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "133abf073a614cf9bfc3c2b6fa986166"
I20260812 02:32:25.939222 23479 raft_consensus.cc:3037] T f003add4ac4749b4a61ff862953ca78e P 133abf073a614cf9bfc3c2b6fa986166 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:32:25.939308 23327 raft_consensus.cc:2454] T f003add4ac4749b4a61ff862953ca78e P 7a91f9c30fb24bc2abf32b1cc6fac06f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5521494881948f68156f1c53716012f in term 1.
I20260812 02:32:25.939399 23646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f003add4ac4749b4a61ff862953ca78e" candidate_uuid: "7a91f9c30fb24bc2abf32b1cc6fac06f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5521494881948f68156f1c53716012f" is_pre_election: true
I20260812 02:32:25.939702 23646 raft_consensus.cc:2379] T f003add4ac4749b4a61ff862953ca78e P a5521494881948f68156f1c53716012f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a91f9c30fb24bc2abf32b1cc6fac06f in current term 1: Already voted for candidate a5521494881948f68156f1c53716012f in this term.
I20260812 02:32:25.939714 23568 leader_election.cc:302] T f003add4ac4749b4a61ff862953ca78e P a5521494881948f68156f1c53716012f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a91f9c30fb24bc2abf32b1cc6fac06f, a5521494881948f68156f1c53716012f; no voters:
I20260812 02:32:25.939915 23754 raft_consensus.cc:2781] T f003add4ac4749b4a61ff862953ca78e P a5521494881948f68156f1c53716012f [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:32:25.940076 23754 raft_consensus.cc:686] T f003add4ac4749b4a61ff862953ca78e P a5521494881948f68156f1c53716012f [term 1 LEADER]: Becoming Leader. State: Replica: a5521494881948f68156f1c53716012f, State: Running, Role: LEADER
I20260812 02:32:25.940157 23480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f003add4ac4749b4a61ff862953ca78e" candidate_uuid: "7a91f9c30fb24bc2abf32b1cc6fac06f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "133abf073a614cf9bfc3c2b6fa986166" is_pre_election: true
I20260812 02:32:25.940431 23754 consensus_queue.cc:237] T f003add4ac4749b4a61ff862953ca78e P a5521494881948f68156f1c53716012f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a91f9c30fb24bc2abf32b1cc6fac06f" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 44325 } } peers { permanent_uuid: "133abf073a614cf9bfc3c2b6fa986166" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 33543 } } peers { permanent_uuid: "a5521494881948f68156f1c53716012f" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 38427 } }
I20260812 02:32:25.940595 23253 leader_election.cc:302] T f003add4ac4749b4a61ff862953ca78e P 7a91f9c30fb24bc2abf32b1cc6fac06f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a91f9c30fb24bc2abf32b1cc6fac06f; no voters: 133abf073a614cf9bfc3c2b6fa986166, a5521494881948f68156f1c53716012f
I20260812 02:32:25.940740 23479 raft_consensus.cc:2454] T f003add4ac4749b4a61ff862953ca78e P 133abf073a614cf9bfc3c2b6fa986166 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5521494881948f68156f1c53716012f in term 1.
I20260812 02:32:25.940976 23739 raft_consensus.cc:2726] T f003add4ac4749b4a61ff862953ca78e P 7a91f9c30fb24bc2abf32b1cc6fac06f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260812 02:32:25.941505 23744 tablet_replica.cc:439] T f003add4ac4749b4a61ff862953ca78e P a5521494881948f68156f1c53716012f: TxnStatusTablet state changed. Reason: New leader a5521494881948f68156f1c53716012f. Latest consensus state: current_term: 1 leader_uuid: "a5521494881948f68156f1c53716012f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a91f9c30fb24bc2abf32b1cc6fac06f" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 44325 } } peers { permanent_uuid: "133abf073a614cf9bfc3c2b6fa986166" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 33543 } } peers { permanent_uuid: "a5521494881948f68156f1c53716012f" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 38427 } } }
I20260812 02:32:25.941636 23744 tablet_replica.cc:442] T f003add4ac4749b4a61ff862953ca78e P a5521494881948f68156f1c53716012f: This TxnStatusTablet replica's current role is: LEADER
I20260812 02:32:25.942016 23778 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260812 02:32:25.942179 23778 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260812 02:32:25.942737 23148 catalog_manager.cc:5495] T f003add4ac4749b4a61ff862953ca78e P a5521494881948f68156f1c53716012f reported cstate change: term changed from 0 to 1, leader changed from <none> to a5521494881948f68156f1c53716012f (127.14.103.3). New cstate: current_term: 1 leader_uuid: "a5521494881948f68156f1c53716012f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a91f9c30fb24bc2abf32b1cc6fac06f" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 44325 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "133abf073a614cf9bfc3c2b6fa986166" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 33543 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5521494881948f68156f1c53716012f" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 38427 } health_report { overall_health: HEALTHY } } }
I20260812 02:32:26.002753 23730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:25.998471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58430 (local address 127.14.103.3:38427)
0812 02:32:25.998629 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:25.998634 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:25.998706 (+ 72us) server_negotiation.cc:407] Connection header received
0812 02:32:25.998886 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:25.998889 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:25.998947 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:25.999072 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 02:32:25.999322 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:26.000110 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:26.001086 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:26.001570 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:26.001691 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:26.002102 (+ 411us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 02:32:26.002208 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 02:32:26.002293 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0812 02:32:26.002531 (+ 238us) server_negotiation.cc:299] Negotiation successful
0812 02:32:26.002600 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260812 02:32:26.002779 23771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:25.998322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.3:38427 (local address 127.0.0.1:58430)
0812 02:32:25.998644 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:25.998657 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 02:32:25.998762 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:32:25.999102 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:32:25.999105 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:32:25.999112 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 02:32:25.999211 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:25.999216 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:26.000273 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:26.000276 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:26.000938 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:26.000948 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:26.001755 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:26.001758 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:26.001857 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:26.001917 (+ 60us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 02:32:26.002334 (+ 417us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 02:32:26.002362 (+ 28us) client_negotiation.cc:769] Sending connection context
0812 02:32:26.002516 (+ 154us) client_negotiation.cc:240] Negotiation successful
0812 02:32:26.002639 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217}
I20260812 02:32:26.005731 23327 raft_consensus.cc:1261] T f003add4ac4749b4a61ff862953ca78e P 7a91f9c30fb24bc2abf32b1cc6fac06f [term 1 FOLLOWER]: Refusing update from remote peer a5521494881948f68156f1c53716012f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:32:26.005771 23479 raft_consensus.cc:1261] T f003add4ac4749b4a61ff862953ca78e P 133abf073a614cf9bfc3c2b6fa986166 [term 1 FOLLOWER]: Refusing update from remote peer a5521494881948f68156f1c53716012f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:32:26.006250 23744 consensus_queue.cc:1035] T f003add4ac4749b4a61ff862953ca78e P a5521494881948f68156f1c53716012f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a91f9c30fb24bc2abf32b1cc6fac06f" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 44325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:32:26.006503 23754 consensus_queue.cc:1035] T f003add4ac4749b4a61ff862953ca78e P a5521494881948f68156f1c53716012f [LEADER]: Connected to new peer: Peer: permanent_uuid: "133abf073a614cf9bfc3c2b6fa986166" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 33543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:32:26.008249 23742 tablet_replica.cc:439] T f003add4ac4749b4a61ff862953ca78e P 133abf073a614cf9bfc3c2b6fa986166: TxnStatusTablet state changed. Reason: New leader a5521494881948f68156f1c53716012f. Latest consensus state: current_term: 1 leader_uuid: "a5521494881948f68156f1c53716012f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a91f9c30fb24bc2abf32b1cc6fac06f" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 44325 } } peers { permanent_uuid: "133abf073a614cf9bfc3c2b6fa986166" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 33543 } } peers { permanent_uuid: "a5521494881948f68156f1c53716012f" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 38427 } } }
I20260812 02:32:26.008221 23739 tablet_replica.cc:439] T f003add4ac4749b4a61ff862953ca78e P 7a91f9c30fb24bc2abf32b1cc6fac06f: TxnStatusTablet state changed. Reason: New leader a5521494881948f68156f1c53716012f. Latest consensus state: current_term: 1 leader_uuid: "a5521494881948f68156f1c53716012f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a91f9c30fb24bc2abf32b1cc6fac06f" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 44325 } } peers { permanent_uuid: "133abf073a614cf9bfc3c2b6fa986166" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 33543 } } peers { permanent_uuid: "a5521494881948f68156f1c53716012f" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 38427 } } }
I20260812 02:32:26.008384 23742 tablet_replica.cc:442] T f003add4ac4749b4a61ff862953ca78e P 133abf073a614cf9bfc3c2b6fa986166: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 02:32:26.008397 23739 tablet_replica.cc:442] T f003add4ac4749b4a61ff862953ca78e P 7a91f9c30fb24bc2abf32b1cc6fac06f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 02:32:26.010552 23754 tablet_replica.cc:439] T f003add4ac4749b4a61ff862953ca78e P a5521494881948f68156f1c53716012f: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a5521494881948f68156f1c53716012f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a91f9c30fb24bc2abf32b1cc6fac06f" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 44325 } } peers { permanent_uuid: "133abf073a614cf9bfc3c2b6fa986166" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 33543 } } peers { permanent_uuid: "a5521494881948f68156f1c53716012f" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 38427 } } }
I20260812 02:32:26.010697 23754 tablet_replica.cc:442] T f003add4ac4749b4a61ff862953ca78e P a5521494881948f68156f1c53716012f: This TxnStatusTablet replica's current role is: LEADER
I20260812 02:32:26.011304 23739 tablet_replica.cc:439] T f003add4ac4749b4a61ff862953ca78e P 7a91f9c30fb24bc2abf32b1cc6fac06f: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a5521494881948f68156f1c53716012f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a91f9c30fb24bc2abf32b1cc6fac06f" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 44325 } } peers { permanent_uuid: "133abf073a614cf9bfc3c2b6fa986166" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 33543 } } peers { permanent_uuid: "a5521494881948f68156f1c53716012f" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 38427 } } }
I20260812 02:32:26.011436 23739 tablet_replica.cc:442] T f003add4ac4749b4a61ff862953ca78e P 7a91f9c30fb24bc2abf32b1cc6fac06f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 02:32:26.011540 23748 mvcc.cc:205] Tried to move back new op lower bound from 7317511970834518016 to 7317511970574798848. Current Snapshot: MvccSnapshot[applied={T|T < 7317511970834518016 or (T in {7317511970834518016})}]
I20260812 02:32:26.016407 23738 tablet_replica.cc:439] T f003add4ac4749b4a61ff862953ca78e P a5521494881948f68156f1c53716012f: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a5521494881948f68156f1c53716012f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a91f9c30fb24bc2abf32b1cc6fac06f" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 44325 } } peers { permanent_uuid: "133abf073a614cf9bfc3c2b6fa986166" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 33543 } } peers { permanent_uuid: "a5521494881948f68156f1c53716012f" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 38427 } } }
I20260812 02:32:26.016566 23738 tablet_replica.cc:442] T f003add4ac4749b4a61ff862953ca78e P a5521494881948f68156f1c53716012f: This TxnStatusTablet replica's current role is: LEADER
I20260812 02:32:26.017623 23751 mvcc.cc:205] Tried to move back new op lower bound from 7317511970834518016 to 7317511970574798848. Current Snapshot: MvccSnapshot[applied={T|T < 7317511970834518016}]
I20260812 02:32:26.018147 23742 tablet_replica.cc:439] T f003add4ac4749b4a61ff862953ca78e P 133abf073a614cf9bfc3c2b6fa986166: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a5521494881948f68156f1c53716012f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a91f9c30fb24bc2abf32b1cc6fac06f" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 44325 } } peers { permanent_uuid: "133abf073a614cf9bfc3c2b6fa986166" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 33543 } } peers { permanent_uuid: "a5521494881948f68156f1c53716012f" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 38427 } } }
I20260812 02:32:26.018266 23742 tablet_replica.cc:442] T f003add4ac4749b4a61ff862953ca78e P 133abf073a614cf9bfc3c2b6fa986166: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 02:32:26.035640 23730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:26.031313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.3:58221 (local address 127.14.103.3:38427)
0812 02:32:26.031495 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:26.031500 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:26.031760 (+ 260us) server_negotiation.cc:407] Connection header received
0812 02:32:26.031899 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:26.031902 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:26.031956 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:26.032030 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 02:32:26.032291 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:26.033094 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:26.034227 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:26.034583 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:26.034687 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:26.035077 (+ 390us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 02:32:26.035171 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 02:32:26.035243 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0812 02:32:26.035422 (+ 179us) server_negotiation.cc:299] Negotiation successful
0812 02:32:26.035481 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260812 02:32:26.035651 23786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:26.031267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.3:38427 (local address 127.14.103.3:58221)
0812 02:32:26.031657 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:26.031671 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 02:32:26.031793 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:32:26.032060 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:32:26.032062 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:32:26.032072 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 02:32:26.032202 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:26.032208 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:26.033229 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:26.033232 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:26.034085 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:26.034097 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:26.034735 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:26.034739 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:26.034837 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:26.034899 (+ 62us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 02:32:26.035273 (+ 374us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 02:32:26.035280 (+ 7us) client_negotiation.cc:769] Sending connection context
0812 02:32:26.035350 (+ 70us) client_negotiation.cc:240] Negotiation successful
0812 02:32:26.035497 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"mutex_wait_us":1,"thread_start_us":103,"threads_started":1}
I20260812 02:32:26.046698 23327 raft_consensus.cc:1261] T d1d02e8e69804f5b96ed8ed55b45b15e P 7a91f9c30fb24bc2abf32b1cc6fac06f [term 1 FOLLOWER]: Refusing update from remote peer a5521494881948f68156f1c53716012f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:32:26.046778 23479 raft_consensus.cc:1261] T d1d02e8e69804f5b96ed8ed55b45b15e P 133abf073a614cf9bfc3c2b6fa986166 [term 1 FOLLOWER]: Refusing update from remote peer a5521494881948f68156f1c53716012f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:32:26.047190 23744 consensus_queue.cc:1035] T d1d02e8e69804f5b96ed8ed55b45b15e P a5521494881948f68156f1c53716012f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a91f9c30fb24bc2abf32b1cc6fac06f" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 44325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:32:26.047504 23744 consensus_queue.cc:1035] T d1d02e8e69804f5b96ed8ed55b45b15e P a5521494881948f68156f1c53716012f [LEADER]: Connected to new peer: Peer: permanent_uuid: "133abf073a614cf9bfc3c2b6fa986166" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 33543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:32:26.090612 23148 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50122:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 02:32:26.090812 23148 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50122:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 02:32:26.093189 23148 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ff23a1d724b647f786e885dc902c2d72: Sending DeleteTablet for 3 replicas of tablet d1d02e8e69804f5b96ed8ed55b45b15e
I20260812 02:32:26.093890 23624 tablet_service.cc:1514] Processing DeleteTablet for tablet d1d02e8e69804f5b96ed8ed55b45b15e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:32:26 UTC) from {username='oryx', principal='oryx/127.14.103.62@KRBTEST.COM'} at 127.0.0.1:58418
I20260812 02:32:26.093900 23460 tablet_service.cc:1514] Processing DeleteTablet for tablet d1d02e8e69804f5b96ed8ed55b45b15e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:32:26 UTC) from {username='oryx', principal='oryx/127.14.103.62@KRBTEST.COM'} at 127.0.0.1:56914
I20260812 02:32:26.093936 23305 tablet_service.cc:1514] Processing DeleteTablet for tablet d1d02e8e69804f5b96ed8ed55b45b15e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:32:26 UTC) from {username='oryx', principal='oryx/127.14.103.62@KRBTEST.COM'} at 127.0.0.1:60422
I20260812 02:32:26.094266 23791 tablet_replica.cc:330] T d1d02e8e69804f5b96ed8ed55b45b15e P 7a91f9c30fb24bc2abf32b1cc6fac06f: stopping tablet replica
I20260812 02:32:26.094460 23791 raft_consensus.cc:2229] T d1d02e8e69804f5b96ed8ed55b45b15e P 7a91f9c30fb24bc2abf32b1cc6fac06f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 02:32:26.094566 23793 tablet_replica.cc:330] T d1d02e8e69804f5b96ed8ed55b45b15e P a5521494881948f68156f1c53716012f: stopping tablet replica
I20260812 02:32:26.094588 23791 raft_consensus.cc:2258] T d1d02e8e69804f5b96ed8ed55b45b15e P 7a91f9c30fb24bc2abf32b1cc6fac06f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:32:26.094686 23793 raft_consensus.cc:2229] T d1d02e8e69804f5b96ed8ed55b45b15e P a5521494881948f68156f1c53716012f [term 1 LEADER]: Raft consensus shutting down.
I20260812 02:32:26.094900 23793 raft_consensus.cc:2258] T d1d02e8e69804f5b96ed8ed55b45b15e P a5521494881948f68156f1c53716012f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:32:26.095443 23791 ts_tablet_manager.cc:1905] T d1d02e8e69804f5b96ed8ed55b45b15e P 7a91f9c30fb24bc2abf32b1cc6fac06f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:32:26.095763 23793 ts_tablet_manager.cc:1905] T d1d02e8e69804f5b96ed8ed55b45b15e P a5521494881948f68156f1c53716012f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:32:26.097661 23792 tablet_replica.cc:330] T d1d02e8e69804f5b96ed8ed55b45b15e P 133abf073a614cf9bfc3c2b6fa986166: stopping tablet replica
I20260812 02:32:26.097831 23792 raft_consensus.cc:2229] T d1d02e8e69804f5b96ed8ed55b45b15e P 133abf073a614cf9bfc3c2b6fa986166 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 02:32:26.097966 23792 raft_consensus.cc:2258] T d1d02e8e69804f5b96ed8ed55b45b15e P 133abf073a614cf9bfc3c2b6fa986166 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:32:26.099023 23793 ts_tablet_manager.cc:1918] T d1d02e8e69804f5b96ed8ed55b45b15e P a5521494881948f68156f1c53716012f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 02:32:26.099109 23793 log.cc:1196] T d1d02e8e69804f5b96ed8ed55b45b15e P a5521494881948f68156f1c53716012f: Deleting WAL directory at /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501898713699-14748-0/minicluster-data/ts-2/wal/wals/d1d02e8e69804f5b96ed8ed55b45b15e
I20260812 02:32:26.099380 23793 ts_tablet_manager.cc:1939] T d1d02e8e69804f5b96ed8ed55b45b15e P a5521494881948f68156f1c53716012f: Deleting consensus metadata
I20260812 02:32:26.100195 23129 catalog_manager.cc:4841] TS a5521494881948f68156f1c53716012f (127.14.103.3:38427): tablet d1d02e8e69804f5b96ed8ed55b45b15e (table test-table [id=a68f004ca989400cb32ad72b04549362]) successfully deleted
I20260812 02:32:26.100656 23792 ts_tablet_manager.cc:1905] T d1d02e8e69804f5b96ed8ed55b45b15e P 133abf073a614cf9bfc3c2b6fa986166: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:32:26.103364 23791 ts_tablet_manager.cc:1918] T d1d02e8e69804f5b96ed8ed55b45b15e P 7a91f9c30fb24bc2abf32b1cc6fac06f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 02:32:26.103578 23791 log.cc:1196] T d1d02e8e69804f5b96ed8ed55b45b15e P 7a91f9c30fb24bc2abf32b1cc6fac06f: Deleting WAL directory at /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501898713699-14748-0/minicluster-data/ts-0/wal/wals/d1d02e8e69804f5b96ed8ed55b45b15e
I20260812 02:32:26.103946 23791 ts_tablet_manager.cc:1939] T d1d02e8e69804f5b96ed8ed55b45b15e P 7a91f9c30fb24bc2abf32b1cc6fac06f: Deleting consensus metadata
I20260812 02:32:26.104571 23792 ts_tablet_manager.cc:1918] T d1d02e8e69804f5b96ed8ed55b45b15e P 133abf073a614cf9bfc3c2b6fa986166: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 02:32:26.104677 23792 log.cc:1196] T d1d02e8e69804f5b96ed8ed55b45b15e P 133abf073a614cf9bfc3c2b6fa986166: Deleting WAL directory at /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501898713699-14748-0/minicluster-data/ts-1/wal/wals/d1d02e8e69804f5b96ed8ed55b45b15e
I20260812 02:32:26.104967 23792 ts_tablet_manager.cc:1939] T d1d02e8e69804f5b96ed8ed55b45b15e P 133abf073a614cf9bfc3c2b6fa986166: Deleting consensus metadata
I20260812 02:32:26.105178 23130 catalog_manager.cc:4841] TS 7a91f9c30fb24bc2abf32b1cc6fac06f (127.14.103.1:44325): tablet d1d02e8e69804f5b96ed8ed55b45b15e (table test-table [id=a68f004ca989400cb32ad72b04549362]) successfully deleted
I20260812 02:32:26.105798 23131 catalog_manager.cc:4841] TS 133abf073a614cf9bfc3c2b6fa986166 (127.14.103.2:33543): tablet d1d02e8e69804f5b96ed8ed55b45b15e (table test-table [id=a68f004ca989400cb32ad72b04549362]) successfully deleted
Aug 12 02:32:26 dist-test-slave-2w3w krb5kdc[23096](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501946, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 12 02:32:26 dist-test-slave-2w3w krb5kdc[23096](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.14.103.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 02:32:26 dist-test-slave-2w3w krb5kdc[23096](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.14.103.62@KRBTEST.COM, Server not found in Kerberos database
W20260812 02:32:26.127985 23203 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:32:26.120110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50146 (local address 127.14.103.62:42935)
0812 02:32:26.120272 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:26.120277 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:26.120507 (+ 230us) server_negotiation.cc:407] Connection header received
0812 02:32:26.120597 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:26.120600 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:26.120648 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:26.120728 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:32:26.121681 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:26.122241 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:26.123385 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:26.123634 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:26.123725 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:26.127824 (+ 4099us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50146: BlockingRecv error: recv got EOF from 127.0.0.1:50146 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Aug 12 02:32:26 dist-test-slave-2w3w krb5kdc[23096](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501946, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.14.103.62@KRBTEST.COM
I20260812 02:32:26.139479 23203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:26.129667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50152 (local address 127.14.103.62:42935)
0812 02:32:26.129847 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:26.129850 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:26.129901 (+ 51us) server_negotiation.cc:407] Connection header received
0812 02:32:26.130056 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:26.130059 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:26.130102 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:26.130166 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:32:26.130892 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:26.131669 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:26.132558 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:26.132816 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:26.132908 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:26.135628 (+ 2720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:32:26.135632 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:32:26.135634 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:32:26.135664 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:32:26.137367 (+ 1703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:26.137827 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:26.137830 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:26.137831 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:26.137946 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:26.138242 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:26.138245 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:26.138246 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:26.138492 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:32:26.138698 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0812 02:32:26.139123 (+ 425us) server_negotiation.cc:299] Negotiation successful
0812 02:32:26.139235 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260812 02:32:26.142895 23148 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50152:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 02:32:26.143445 23148 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 02:32:26.148867 23305 tablet_service.cc:1467] Processing CreateTablet for tablet 46e6c7dd66f74bbda6d1dbce15ce1048 (DEFAULT_TABLE table=test-table [id=e9fcbaf297dd4699a60fc5b1e283f2a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:32:26.148964 23624 tablet_service.cc:1467] Processing CreateTablet for tablet 46e6c7dd66f74bbda6d1dbce15ce1048 (DEFAULT_TABLE table=test-table [id=e9fcbaf297dd4699a60fc5b1e283f2a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:32:26.148929 23460 tablet_service.cc:1467] Processing CreateTablet for tablet 46e6c7dd66f74bbda6d1dbce15ce1048 (DEFAULT_TABLE table=test-table [id=e9fcbaf297dd4699a60fc5b1e283f2a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:32:26.149257 23624 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46e6c7dd66f74bbda6d1dbce15ce1048. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:32:26.149255 23305 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46e6c7dd66f74bbda6d1dbce15ce1048. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:32:26.149268 23460 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46e6c7dd66f74bbda6d1dbce15ce1048. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:32:26.151726 23733 tablet_bootstrap.cc:492] T 46e6c7dd66f74bbda6d1dbce15ce1048 P 133abf073a614cf9bfc3c2b6fa986166: Bootstrap starting.
I20260812 02:32:26.151832 23731 tablet_bootstrap.cc:492] T 46e6c7dd66f74bbda6d1dbce15ce1048 P 7a91f9c30fb24bc2abf32b1cc6fac06f: Bootstrap starting.
I20260812 02:32:26.151929 23734 tablet_bootstrap.cc:492] T 46e6c7dd66f74bbda6d1dbce15ce1048 P a5521494881948f68156f1c53716012f: Bootstrap starting.
I20260812 02:32:26.152850 23731 tablet_bootstrap.cc:654] T 46e6c7dd66f74bbda6d1dbce15ce1048 P 7a91f9c30fb24bc2abf32b1cc6fac06f: Neither blocks nor log segments found. Creating new log.
I20260812 02:32:26.152850 23734 tablet_bootstrap.cc:654] T 46e6c7dd66f74bbda6d1dbce15ce1048 P a5521494881948f68156f1c53716012f: Neither blocks nor log segments found. Creating new log.
I20260812 02:32:26.152869 23733 tablet_bootstrap.cc:654] T 46e6c7dd66f74bbda6d1dbce15ce1048 P 133abf073a614cf9bfc3c2b6fa986166: Neither blocks nor log segments found. Creating new log.
I20260812 02:32:26.154382 23731 tablet_bootstrap.cc:492] T 46e6c7dd66f74bbda6d1dbce15ce1048 P 7a91f9c30fb24bc2abf32b1cc6fac06f: No bootstrap required, opened a new log
I20260812 02:32:26.154495 23731 ts_tablet_manager.cc:1397] T 46e6c7dd66f74bbda6d1dbce15ce1048 P 7a91f9c30fb24bc2abf32b1cc6fac06f: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260812 02:32:26.154493 23733 tablet_bootstrap.cc:492] T 46e6c7dd66f74bbda6d1dbce15ce1048 P 133abf073a614cf9bfc3c2b6fa986166: No bootstrap required, opened a new log
I20260812 02:32:26.154587 23734 tablet_bootstrap.cc:492] T 46e6c7dd66f74bbda6d1dbce15ce1048 P a5521494881948f68156f1c53716012f: No bootstrap required, opened a new log
I20260812 02:32:26.154588 23733 ts_tablet_manager.cc:1397] T 46e6c7dd66f74bbda6d1dbce15ce1048 P 133abf073a614cf9bfc3c2b6fa986166: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260812 02:32:26.154693 23734 ts_tablet_manager.cc:1397] T 46e6c7dd66f74bbda6d1dbce15ce1048 P a5521494881948f68156f1c53716012f: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260812 02:32:26.155086 23731 raft_consensus.cc:348] T 46e6c7dd66f74bbda6d1dbce15ce1048 P 7a91f9c30fb24bc2abf32b1cc6fac06f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a91f9c30fb24bc2abf32b1cc6fac06f" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 44325 } } peers { permanent_uuid: "133abf073a614cf9bfc3c2b6fa986166" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 33543 } } peers { permanent_uuid: "a5521494881948f68156f1c53716012f" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 38427 } }
I20260812 02:32:26.155118 23734 raft_consensus.cc:348] T 46e6c7dd66f74bbda6d1dbce15ce1048 P a5521494881948f68156f1c53716012f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a91f9c30fb24bc2abf32b1cc6fac06f" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 44325 } } peers { permanent_uuid: "133abf073a614cf9bfc3c2b6fa986166" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 33543 } } peers { permanent_uuid: "a5521494881948f68156f1c53716012f" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 38427 } }
I20260812 02:32:26.155231 23731 raft_consensus.cc:374] T 46e6c7dd66f74bbda6d1dbce15ce1048 P 7a91f9c30fb24bc2abf32b1cc6fac06f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:32:26.155231 23734 raft_consensus.cc:374] T 46e6c7dd66f74bbda6d1dbce15ce1048 P a5521494881948f68156f1c53716012f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:32:26.155267 23734 raft_consensus.cc:729] T 46e6c7dd66f74bbda6d1dbce15ce1048 P a5521494881948f68156f1c53716012f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5521494881948f68156f1c53716012f, State: Initialized, Role: FOLLOWER
I20260812 02:32:26.155267 23731 raft_consensus.cc:729] T 46e6c7dd66f74bbda6d1dbce15ce1048 P 7a91f9c30fb24bc2abf32b1cc6fac06f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a91f9c30fb24bc2abf32b1cc6fac06f, State: Initialized, Role: FOLLOWER
I20260812 02:32:26.155401 23731 consensus_queue.cc:260] T 46e6c7dd66f74bbda6d1dbce15ce1048 P 7a91f9c30fb24bc2abf32b1cc6fac06f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a91f9c30fb24bc2abf32b1cc6fac06f" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 44325 } } peers { permanent_uuid: "133abf073a614cf9bfc3c2b6fa986166" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 33543 } } peers { permanent_uuid: "a5521494881948f68156f1c53716012f" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 38427 } }
I20260812 02:32:26.155401 23734 consensus_queue.cc:260] T 46e6c7dd66f74bbda6d1dbce15ce1048 P a5521494881948f68156f1c53716012f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a91f9c30fb24bc2abf32b1cc6fac06f" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 44325 } } peers { permanent_uuid: "133abf073a614cf9bfc3c2b6fa986166" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 33543 } } peers { permanent_uuid: "a5521494881948f68156f1c53716012f" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 38427 } }
I20260812 02:32:26.155475 23733 raft_consensus.cc:348] T 46e6c7dd66f74bbda6d1dbce15ce1048 P 133abf073a614cf9bfc3c2b6fa986166 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a91f9c30fb24bc2abf32b1cc6fac06f" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 44325 } } peers { permanent_uuid: "133abf073a614cf9bfc3c2b6fa986166" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 33543 } } peers { permanent_uuid: "a5521494881948f68156f1c53716012f" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 38427 } }
I20260812 02:32:26.155637 23733 raft_consensus.cc:374] T 46e6c7dd66f74bbda6d1dbce15ce1048 P 133abf073a614cf9bfc3c2b6fa986166 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:32:26.155678 23734 ts_tablet_manager.cc:1428] T 46e6c7dd66f74bbda6d1dbce15ce1048 P a5521494881948f68156f1c53716012f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260812 02:32:26.155678 23731 ts_tablet_manager.cc:1428] T 46e6c7dd66f74bbda6d1dbce15ce1048 P 7a91f9c30fb24bc2abf32b1cc6fac06f: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260812 02:32:26.155681 23733 raft_consensus.cc:729] T 46e6c7dd66f74bbda6d1dbce15ce1048 P 133abf073a614cf9bfc3c2b6fa986166 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 133abf073a614cf9bfc3c2b6fa986166, State: Initialized, Role: FOLLOWER
I20260812 02:32:26.155827 23733 consensus_queue.cc:260] T 46e6c7dd66f74bbda6d1dbce15ce1048 P 133abf073a614cf9bfc3c2b6fa986166 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a91f9c30fb24bc2abf32b1cc6fac06f" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 44325 } } peers { permanent_uuid: "133abf073a614cf9bfc3c2b6fa986166" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 33543 } } peers { permanent_uuid: "a5521494881948f68156f1c53716012f" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 38427 } }
I20260812 02:32:26.156109 23733 ts_tablet_manager.cc:1428] T 46e6c7dd66f74bbda6d1dbce15ce1048 P 133abf073a614cf9bfc3c2b6fa986166: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260812 02:32:26.272583 23739 raft_consensus.cc:482] T 46e6c7dd66f74bbda6d1dbce15ce1048 P 7a91f9c30fb24bc2abf32b1cc6fac06f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 02:32:26.272748 23739 raft_consensus.cc:504] T 46e6c7dd66f74bbda6d1dbce15ce1048 P 7a91f9c30fb24bc2abf32b1cc6fac06f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a91f9c30fb24bc2abf32b1cc6fac06f" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 44325 } } peers { permanent_uuid: "133abf073a614cf9bfc3c2b6fa986166" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 33543 } } peers { permanent_uuid: "a5521494881948f68156f1c53716012f" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 38427 } }
I20260812 02:32:26.273176 23739 leader_election.cc:288] T 46e6c7dd66f74bbda6d1dbce15ce1048 P 7a91f9c30fb24bc2abf32b1cc6fac06f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 133abf073a614cf9bfc3c2b6fa986166 (127.14.103.2:33543), a5521494881948f68156f1c53716012f (127.14.103.3:38427)
I20260812 02:32:26.273859 23479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46e6c7dd66f74bbda6d1dbce15ce1048" candidate_uuid: "7a91f9c30fb24bc2abf32b1cc6fac06f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "133abf073a614cf9bfc3c2b6fa986166" is_pre_election: true
I20260812 02:32:26.273885 23646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46e6c7dd66f74bbda6d1dbce15ce1048" candidate_uuid: "7a91f9c30fb24bc2abf32b1cc6fac06f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5521494881948f68156f1c53716012f" is_pre_election: true
I20260812 02:32:26.274057 23479 raft_consensus.cc:2454] T 46e6c7dd66f74bbda6d1dbce15ce1048 P 133abf073a614cf9bfc3c2b6fa986166 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a91f9c30fb24bc2abf32b1cc6fac06f in term 0.
I20260812 02:32:26.274116 23646 raft_consensus.cc:2454] T 46e6c7dd66f74bbda6d1dbce15ce1048 P a5521494881948f68156f1c53716012f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a91f9c30fb24bc2abf32b1cc6fac06f in term 0.
I20260812 02:32:26.274478 23253 leader_election.cc:302] T 46e6c7dd66f74bbda6d1dbce15ce1048 P 7a91f9c30fb24bc2abf32b1cc6fac06f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 133abf073a614cf9bfc3c2b6fa986166, 7a91f9c30fb24bc2abf32b1cc6fac06f; no voters:
I20260812 02:32:26.274698 23739 raft_consensus.cc:2781] T 46e6c7dd66f74bbda6d1dbce15ce1048 P 7a91f9c30fb24bc2abf32b1cc6fac06f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 02:32:26.274775 23739 raft_consensus.cc:482] T 46e6c7dd66f74bbda6d1dbce15ce1048 P 7a91f9c30fb24bc2abf32b1cc6fac06f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 02:32:26.274830 23739 raft_consensus.cc:3037] T 46e6c7dd66f74bbda6d1dbce15ce1048 P 7a91f9c30fb24bc2abf32b1cc6fac06f [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:32:26.276054 23739 raft_consensus.cc:504] T 46e6c7dd66f74bbda6d1dbce15ce1048 P 7a91f9c30fb24bc2abf32b1cc6fac06f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a91f9c30fb24bc2abf32b1cc6fac06f" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 44325 } } peers { permanent_uuid: "133abf073a614cf9bfc3c2b6fa986166" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 33543 } } peers { permanent_uuid: "a5521494881948f68156f1c53716012f" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 38427 } }
I20260812 02:32:26.276510 23739 leader_election.cc:288] T 46e6c7dd66f74bbda6d1dbce15ce1048 P 7a91f9c30fb24bc2abf32b1cc6fac06f [CANDIDATE]: Term 1 election: Requested vote from peers 133abf073a614cf9bfc3c2b6fa986166 (127.14.103.2:33543), a5521494881948f68156f1c53716012f (127.14.103.3:38427)
I20260812 02:32:26.276997 23479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46e6c7dd66f74bbda6d1dbce15ce1048" candidate_uuid: "7a91f9c30fb24bc2abf32b1cc6fac06f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "133abf073a614cf9bfc3c2b6fa986166"
I20260812 02:32:26.277065 23646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46e6c7dd66f74bbda6d1dbce15ce1048" candidate_uuid: "7a91f9c30fb24bc2abf32b1cc6fac06f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5521494881948f68156f1c53716012f"
I20260812 02:32:26.277166 23479 raft_consensus.cc:3037] T 46e6c7dd66f74bbda6d1dbce15ce1048 P 133abf073a614cf9bfc3c2b6fa986166 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:32:26.277223 23646 raft_consensus.cc:3037] T 46e6c7dd66f74bbda6d1dbce15ce1048 P a5521494881948f68156f1c53716012f [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:32:26.278287 23646 raft_consensus.cc:2454] T 46e6c7dd66f74bbda6d1dbce15ce1048 P a5521494881948f68156f1c53716012f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a91f9c30fb24bc2abf32b1cc6fac06f in term 1.
I20260812 02:32:26.278388 23479 raft_consensus.cc:2454] T 46e6c7dd66f74bbda6d1dbce15ce1048 P 133abf073a614cf9bfc3c2b6fa986166 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a91f9c30fb24bc2abf32b1cc6fac06f in term 1.
I20260812 02:32:26.278728 23251 leader_election.cc:302] T 46e6c7dd66f74bbda6d1dbce15ce1048 P 7a91f9c30fb24bc2abf32b1cc6fac06f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a91f9c30fb24bc2abf32b1cc6fac06f, a5521494881948f68156f1c53716012f; no voters:
I20260812 02:32:26.278992 23739 raft_consensus.cc:2781] T 46e6c7dd66f74bbda6d1dbce15ce1048 P 7a91f9c30fb24bc2abf32b1cc6fac06f [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:32:26.279213 23739 raft_consensus.cc:686] T 46e6c7dd66f74bbda6d1dbce15ce1048 P 7a91f9c30fb24bc2abf32b1cc6fac06f [term 1 LEADER]: Becoming Leader. State: Replica: 7a91f9c30fb24bc2abf32b1cc6fac06f, State: Running, Role: LEADER
I20260812 02:32:26.279537 23739 consensus_queue.cc:237] T 46e6c7dd66f74bbda6d1dbce15ce1048 P 7a91f9c30fb24bc2abf32b1cc6fac06f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a91f9c30fb24bc2abf32b1cc6fac06f" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 44325 } } peers { permanent_uuid: "133abf073a614cf9bfc3c2b6fa986166" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 33543 } } peers { permanent_uuid: "a5521494881948f68156f1c53716012f" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 38427 } }
I20260812 02:32:26.282138 23147 catalog_manager.cc:5495] T 46e6c7dd66f74bbda6d1dbce15ce1048 P 7a91f9c30fb24bc2abf32b1cc6fac06f reported cstate change: term changed from 0 to 1, leader changed from <none> to 7a91f9c30fb24bc2abf32b1cc6fac06f (127.14.103.1). New cstate: current_term: 1 leader_uuid: "7a91f9c30fb24bc2abf32b1cc6fac06f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a91f9c30fb24bc2abf32b1cc6fac06f" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 44325 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "133abf073a614cf9bfc3c2b6fa986166" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 33543 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5521494881948f68156f1c53716012f" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 38427 } health_report { overall_health: UNKNOWN } } }
I20260812 02:32:26.321861 23817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:26.317798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60430 (local address 127.14.103.1:44325)
0812 02:32:26.318113 (+ 315us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:26.318120 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:26.318132 (+ 12us) server_negotiation.cc:407] Connection header received
0812 02:32:26.318217 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:26.318219 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:26.318274 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:26.318413 (+ 139us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 02:32:26.318692 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:26.319308 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:26.320222 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:26.320475 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:26.320591 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:26.320896 (+ 305us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 02:32:26.321403 (+ 507us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 02:32:26.321542 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0812 02:32:26.321605 (+ 63us) server_negotiation.cc:299] Negotiation successful
0812 02:32:26.321672 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":246,"thread_start_us":117,"threads_started":1}
I20260812 02:32:26.325178 23479 raft_consensus.cc:1261] T 46e6c7dd66f74bbda6d1dbce15ce1048 P 133abf073a614cf9bfc3c2b6fa986166 [term 1 FOLLOWER]: Refusing update from remote peer 7a91f9c30fb24bc2abf32b1cc6fac06f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:32:26.325282 23646 raft_consensus.cc:1261] T 46e6c7dd66f74bbda6d1dbce15ce1048 P a5521494881948f68156f1c53716012f [term 1 FOLLOWER]: Refusing update from remote peer 7a91f9c30fb24bc2abf32b1cc6fac06f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:32:26.325841 23812 consensus_queue.cc:1035] T 46e6c7dd66f74bbda6d1dbce15ce1048 P 7a91f9c30fb24bc2abf32b1cc6fac06f [LEADER]: Connected to new peer: Peer: permanent_uuid: "133abf073a614cf9bfc3c2b6fa986166" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 33543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:32:26.326097 23739 consensus_queue.cc:1035] T 46e6c7dd66f74bbda6d1dbce15ce1048 P 7a91f9c30fb24bc2abf32b1cc6fac06f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5521494881948f68156f1c53716012f" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 38427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:32:26.339654 23146 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50152:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 02:32:26.339854 23146 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50152:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 02:32:26.342137 23146 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ff23a1d724b647f786e885dc902c2d72: Sending DeleteTablet for 3 replicas of tablet 46e6c7dd66f74bbda6d1dbce15ce1048
I20260812 02:32:26.342911 23305 tablet_service.cc:1514] Processing DeleteTablet for tablet 46e6c7dd66f74bbda6d1dbce15ce1048 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:32:26 UTC) from {username='oryx', principal='oryx/127.14.103.62@KRBTEST.COM'} at 127.0.0.1:60422
I20260812 02:32:26.342911 23460 tablet_service.cc:1514] Processing DeleteTablet for tablet 46e6c7dd66f74bbda6d1dbce15ce1048 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:32:26 UTC) from {username='oryx', principal='oryx/127.14.103.62@KRBTEST.COM'} at 127.0.0.1:56914
I20260812 02:32:26.343137 23792 tablet_replica.cc:330] T 46e6c7dd66f74bbda6d1dbce15ce1048 P 133abf073a614cf9bfc3c2b6fa986166: stopping tablet replica
I20260812 02:32:26.343137 23791 tablet_replica.cc:330] T 46e6c7dd66f74bbda6d1dbce15ce1048 P 7a91f9c30fb24bc2abf32b1cc6fac06f: stopping tablet replica
I20260812 02:32:26.343255 23791 raft_consensus.cc:2229] T 46e6c7dd66f74bbda6d1dbce15ce1048 P 7a91f9c30fb24bc2abf32b1cc6fac06f [term 1 LEADER]: Raft consensus shutting down.
I20260812 02:32:26.343255 23792 raft_consensus.cc:2229] T 46e6c7dd66f74bbda6d1dbce15ce1048 P 133abf073a614cf9bfc3c2b6fa986166 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 02:32:26.343291 23624 tablet_service.cc:1514] Processing DeleteTablet for tablet 46e6c7dd66f74bbda6d1dbce15ce1048 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:32:26 UTC) from {username='oryx', principal='oryx/127.14.103.62@KRBTEST.COM'} at 127.0.0.1:58418
I20260812 02:32:26.343389 23792 raft_consensus.cc:2258] T 46e6c7dd66f74bbda6d1dbce15ce1048 P 133abf073a614cf9bfc3c2b6fa986166 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:32:26.343515 23793 tablet_replica.cc:330] T 46e6c7dd66f74bbda6d1dbce15ce1048 P a5521494881948f68156f1c53716012f: stopping tablet replica
I20260812 02:32:26.343575 23791 raft_consensus.cc:2258] T 46e6c7dd66f74bbda6d1dbce15ce1048 P 7a91f9c30fb24bc2abf32b1cc6fac06f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:32:26.343641 23793 raft_consensus.cc:2229] T 46e6c7dd66f74bbda6d1dbce15ce1048 P a5521494881948f68156f1c53716012f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 02:32:26.343781 23793 raft_consensus.cc:2258] T 46e6c7dd66f74bbda6d1dbce15ce1048 P a5521494881948f68156f1c53716012f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:32:26.344729 23791 ts_tablet_manager.cc:1905] T 46e6c7dd66f74bbda6d1dbce15ce1048 P 7a91f9c30fb24bc2abf32b1cc6fac06f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:32:26.344795 23146 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50152:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260812 02:32:26.345170 23793 ts_tablet_manager.cc:1905] T 46e6c7dd66f74bbda6d1dbce15ce1048 P a5521494881948f68156f1c53716012f: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260812 02:32:26.345332 23146 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 02:32:26.347692 23791 ts_tablet_manager.cc:1918] T 46e6c7dd66f74bbda6d1dbce15ce1048 P 7a91f9c30fb24bc2abf32b1cc6fac06f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 02:32:26.347760 23793 ts_tablet_manager.cc:1918] T 46e6c7dd66f74bbda6d1dbce15ce1048 P a5521494881948f68156f1c53716012f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 02:32:26.347784 23791 log.cc:1196] T 46e6c7dd66f74bbda6d1dbce15ce1048 P 7a91f9c30fb24bc2abf32b1cc6fac06f: Deleting WAL directory at /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501898713699-14748-0/minicluster-data/ts-0/wal/wals/46e6c7dd66f74bbda6d1dbce15ce1048
I20260812 02:32:26.347831 23793 log.cc:1196] T 46e6c7dd66f74bbda6d1dbce15ce1048 P a5521494881948f68156f1c53716012f: Deleting WAL directory at /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501898713699-14748-0/minicluster-data/ts-2/wal/wals/46e6c7dd66f74bbda6d1dbce15ce1048
I20260812 02:32:26.348068 23791 ts_tablet_manager.cc:1939] T 46e6c7dd66f74bbda6d1dbce15ce1048 P 7a91f9c30fb24bc2abf32b1cc6fac06f: Deleting consensus metadata
I20260812 02:32:26.348104 23793 ts_tablet_manager.cc:1939] T 46e6c7dd66f74bbda6d1dbce15ce1048 P a5521494881948f68156f1c53716012f: Deleting consensus metadata
I20260812 02:32:26.349596 23792 ts_tablet_manager.cc:1905] T 46e6c7dd66f74bbda6d1dbce15ce1048 P 133abf073a614cf9bfc3c2b6fa986166: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:32:26.349574 23129 catalog_manager.cc:4841] TS a5521494881948f68156f1c53716012f (127.14.103.3:38427): tablet 46e6c7dd66f74bbda6d1dbce15ce1048 (table test-table [id=e9fcbaf297dd4699a60fc5b1e283f2a9]) successfully deleted
I20260812 02:32:26.350073 23130 catalog_manager.cc:4841] TS 7a91f9c30fb24bc2abf32b1cc6fac06f (127.14.103.1:44325): tablet 46e6c7dd66f74bbda6d1dbce15ce1048 (table test-table [id=e9fcbaf297dd4699a60fc5b1e283f2a9]) successfully deleted
I20260812 02:32:26.352504 23792 ts_tablet_manager.cc:1918] T 46e6c7dd66f74bbda6d1dbce15ce1048 P 133abf073a614cf9bfc3c2b6fa986166: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 02:32:26.352612 23792 log.cc:1196] T 46e6c7dd66f74bbda6d1dbce15ce1048 P 133abf073a614cf9bfc3c2b6fa986166: Deleting WAL directory at /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501898713699-14748-0/minicluster-data/ts-1/wal/wals/46e6c7dd66f74bbda6d1dbce15ce1048
I20260812 02:32:26.352895 23792 ts_tablet_manager.cc:1939] T 46e6c7dd66f74bbda6d1dbce15ce1048 P 133abf073a614cf9bfc3c2b6fa986166: Deleting consensus metadata
I20260812 02:32:26.353288 23460 tablet_service.cc:1467] Processing CreateTablet for tablet d0fac415b6bf4b1284bb7afcacf6badb (DEFAULT_TABLE table=test-table [id=d0256955a73c4d4cb6d6dc0fa6ce3dc5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:32:26.353569 23460 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0fac415b6bf4b1284bb7afcacf6badb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:32:26.353693 23131 catalog_manager.cc:4841] TS 133abf073a614cf9bfc3c2b6fa986166 (127.14.103.2:33543): tablet 46e6c7dd66f74bbda6d1dbce15ce1048 (table test-table [id=e9fcbaf297dd4699a60fc5b1e283f2a9]) successfully deleted
I20260812 02:32:26.355547 23733 tablet_bootstrap.cc:492] T d0fac415b6bf4b1284bb7afcacf6badb P 133abf073a614cf9bfc3c2b6fa986166: Bootstrap starting.
I20260812 02:32:26.355525 23305 tablet_service.cc:1467] Processing CreateTablet for tablet d0fac415b6bf4b1284bb7afcacf6badb (DEFAULT_TABLE table=test-table [id=d0256955a73c4d4cb6d6dc0fa6ce3dc5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:32:26.355799 23305 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0fac415b6bf4b1284bb7afcacf6badb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:32:26.355988 23624 tablet_service.cc:1467] Processing CreateTablet for tablet d0fac415b6bf4b1284bb7afcacf6badb (DEFAULT_TABLE table=test-table [id=d0256955a73c4d4cb6d6dc0fa6ce3dc5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:32:26.356290 23624 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0fac415b6bf4b1284bb7afcacf6badb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:32:26.356591 23733 tablet_bootstrap.cc:654] T d0fac415b6bf4b1284bb7afcacf6badb P 133abf073a614cf9bfc3c2b6fa986166: Neither blocks nor log segments found. Creating new log.
I20260812 02:32:26.358353 23731 tablet_bootstrap.cc:492] T d0fac415b6bf4b1284bb7afcacf6badb P 7a91f9c30fb24bc2abf32b1cc6fac06f: Bootstrap starting.
I20260812 02:32:26.358686 23734 tablet_bootstrap.cc:492] T d0fac415b6bf4b1284bb7afcacf6badb P a5521494881948f68156f1c53716012f: Bootstrap starting.
I20260812 02:32:26.359140 23733 tablet_bootstrap.cc:492] T d0fac415b6bf4b1284bb7afcacf6badb P 133abf073a614cf9bfc3c2b6fa986166: No bootstrap required, opened a new log
I20260812 02:32:26.359236 23733 ts_tablet_manager.cc:1397] T d0fac415b6bf4b1284bb7afcacf6badb P 133abf073a614cf9bfc3c2b6fa986166: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260812 02:32:26.359499 23731 tablet_bootstrap.cc:654] T d0fac415b6bf4b1284bb7afcacf6badb P 7a91f9c30fb24bc2abf32b1cc6fac06f: Neither blocks nor log segments found. Creating new log.
I20260812 02:32:26.359630 23734 tablet_bootstrap.cc:654] T d0fac415b6bf4b1284bb7afcacf6badb P a5521494881948f68156f1c53716012f: Neither blocks nor log segments found. Creating new log.
I20260812 02:32:26.359812 23733 raft_consensus.cc:348] T d0fac415b6bf4b1284bb7afcacf6badb P 133abf073a614cf9bfc3c2b6fa986166 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a91f9c30fb24bc2abf32b1cc6fac06f" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 44325 } } peers { permanent_uuid: "133abf073a614cf9bfc3c2b6fa986166" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 33543 } } peers { permanent_uuid: "a5521494881948f68156f1c53716012f" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 38427 } }
I20260812 02:32:26.359970 23733 raft_consensus.cc:374] T d0fac415b6bf4b1284bb7afcacf6badb P 133abf073a614cf9bfc3c2b6fa986166 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:32:26.360010 23733 raft_consensus.cc:729] T d0fac415b6bf4b1284bb7afcacf6badb P 133abf073a614cf9bfc3c2b6fa986166 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 133abf073a614cf9bfc3c2b6fa986166, State: Initialized, Role: FOLLOWER
I20260812 02:32:26.360159 23733 consensus_queue.cc:260] T d0fac415b6bf4b1284bb7afcacf6badb P 133abf073a614cf9bfc3c2b6fa986166 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a91f9c30fb24bc2abf32b1cc6fac06f" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 44325 } } peers { permanent_uuid: "133abf073a614cf9bfc3c2b6fa986166" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 33543 } } peers { permanent_uuid: "a5521494881948f68156f1c53716012f" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 38427 } }
I20260812 02:32:26.360453 23733 ts_tablet_manager.cc:1428] T d0fac415b6bf4b1284bb7afcacf6badb P 133abf073a614cf9bfc3c2b6fa986166: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260812 02:32:26.360937 23731 tablet_bootstrap.cc:492] T d0fac415b6bf4b1284bb7afcacf6badb P 7a91f9c30fb24bc2abf32b1cc6fac06f: No bootstrap required, opened a new log
I20260812 02:32:26.361043 23731 ts_tablet_manager.cc:1397] T d0fac415b6bf4b1284bb7afcacf6badb P 7a91f9c30fb24bc2abf32b1cc6fac06f: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260812 02:32:26.361685 23731 raft_consensus.cc:348] T d0fac415b6bf4b1284bb7afcacf6badb P 7a91f9c30fb24bc2abf32b1cc6fac06f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a91f9c30fb24bc2abf32b1cc6fac06f" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 44325 } } peers { permanent_uuid: "133abf073a614cf9bfc3c2b6fa986166" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 33543 } } peers { permanent_uuid: "a5521494881948f68156f1c53716012f" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 38427 } }
I20260812 02:32:26.361835 23731 raft_consensus.cc:374] T d0fac415b6bf4b1284bb7afcacf6badb P 7a91f9c30fb24bc2abf32b1cc6fac06f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:32:26.361891 23731 raft_consensus.cc:729] T d0fac415b6bf4b1284bb7afcacf6badb P 7a91f9c30fb24bc2abf32b1cc6fac06f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a91f9c30fb24bc2abf32b1cc6fac06f, State: Initialized, Role: FOLLOWER
I20260812 02:32:26.362052 23731 consensus_queue.cc:260] T d0fac415b6bf4b1284bb7afcacf6badb P 7a91f9c30fb24bc2abf32b1cc6fac06f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a91f9c30fb24bc2abf32b1cc6fac06f" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 44325 } } peers { permanent_uuid: "133abf073a614cf9bfc3c2b6fa986166" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 33543 } } peers { permanent_uuid: "a5521494881948f68156f1c53716012f" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 38427 } }
I20260812 02:32:26.362373 23731 ts_tablet_manager.cc:1428] T d0fac415b6bf4b1284bb7afcacf6badb P 7a91f9c30fb24bc2abf32b1cc6fac06f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260812 02:32:26.372275 23734 tablet_bootstrap.cc:492] T d0fac415b6bf4b1284bb7afcacf6badb P a5521494881948f68156f1c53716012f: No bootstrap required, opened a new log
I20260812 02:32:26.372399 23734 ts_tablet_manager.cc:1397] T d0fac415b6bf4b1284bb7afcacf6badb P a5521494881948f68156f1c53716012f: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.000s
I20260812 02:32:26.372885 23734 raft_consensus.cc:348] T d0fac415b6bf4b1284bb7afcacf6badb P a5521494881948f68156f1c53716012f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a91f9c30fb24bc2abf32b1cc6fac06f" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 44325 } } peers { permanent_uuid: "133abf073a614cf9bfc3c2b6fa986166" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 33543 } } peers { permanent_uuid: "a5521494881948f68156f1c53716012f" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 38427 } }
I20260812 02:32:26.373029 23734 raft_consensus.cc:374] T d0fac415b6bf4b1284bb7afcacf6badb P a5521494881948f68156f1c53716012f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:32:26.373101 23734 raft_consensus.cc:729] T d0fac415b6bf4b1284bb7afcacf6badb P a5521494881948f68156f1c53716012f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5521494881948f68156f1c53716012f, State: Initialized, Role: FOLLOWER
I20260812 02:32:26.373294 23734 consensus_queue.cc:260] T d0fac415b6bf4b1284bb7afcacf6badb P a5521494881948f68156f1c53716012f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a91f9c30fb24bc2abf32b1cc6fac06f" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 44325 } } peers { permanent_uuid: "133abf073a614cf9bfc3c2b6fa986166" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 33543 } } peers { permanent_uuid: "a5521494881948f68156f1c53716012f" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 38427 } }
I20260812 02:32:26.373620 23734 ts_tablet_manager.cc:1428] T d0fac415b6bf4b1284bb7afcacf6badb P a5521494881948f68156f1c53716012f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260812 02:32:26.379658 23744 raft_consensus.cc:482] T d0fac415b6bf4b1284bb7afcacf6badb P a5521494881948f68156f1c53716012f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 02:32:26.379863 23744 raft_consensus.cc:504] T d0fac415b6bf4b1284bb7afcacf6badb P a5521494881948f68156f1c53716012f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a91f9c30fb24bc2abf32b1cc6fac06f" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 44325 } } peers { permanent_uuid: "133abf073a614cf9bfc3c2b6fa986166" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 33543 } } peers { permanent_uuid: "a5521494881948f68156f1c53716012f" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 38427 } }
I20260812 02:32:26.380450 23744 leader_election.cc:288] T d0fac415b6bf4b1284bb7afcacf6badb P a5521494881948f68156f1c53716012f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a91f9c30fb24bc2abf32b1cc6fac06f (127.14.103.1:44325), 133abf073a614cf9bfc3c2b6fa986166 (127.14.103.2:33543)
I20260812 02:32:26.380997 23327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0fac415b6bf4b1284bb7afcacf6badb" candidate_uuid: "a5521494881948f68156f1c53716012f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a91f9c30fb24bc2abf32b1cc6fac06f" is_pre_election: true
I20260812 02:32:26.381143 23327 raft_consensus.cc:2454] T d0fac415b6bf4b1284bb7afcacf6badb P 7a91f9c30fb24bc2abf32b1cc6fac06f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5521494881948f68156f1c53716012f in term 0.
I20260812 02:32:26.381448 23479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0fac415b6bf4b1284bb7afcacf6badb" candidate_uuid: "a5521494881948f68156f1c53716012f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "133abf073a614cf9bfc3c2b6fa986166" is_pre_election: true
I20260812 02:32:26.381497 23568 leader_election.cc:302] T d0fac415b6bf4b1284bb7afcacf6badb P a5521494881948f68156f1c53716012f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a91f9c30fb24bc2abf32b1cc6fac06f, a5521494881948f68156f1c53716012f; no voters:
I20260812 02:32:26.381608 23479 raft_consensus.cc:2454] T d0fac415b6bf4b1284bb7afcacf6badb P 133abf073a614cf9bfc3c2b6fa986166 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5521494881948f68156f1c53716012f in term 0.
I20260812 02:32:26.381686 23744 raft_consensus.cc:2781] T d0fac415b6bf4b1284bb7afcacf6badb P a5521494881948f68156f1c53716012f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 02:32:26.381740 23744 raft_consensus.cc:482] T d0fac415b6bf4b1284bb7afcacf6badb P a5521494881948f68156f1c53716012f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 02:32:26.381788 23744 raft_consensus.cc:3037] T d0fac415b6bf4b1284bb7afcacf6badb P a5521494881948f68156f1c53716012f [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:32:26.383055 23744 raft_consensus.cc:504] T d0fac415b6bf4b1284bb7afcacf6badb P a5521494881948f68156f1c53716012f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a91f9c30fb24bc2abf32b1cc6fac06f" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 44325 } } peers { permanent_uuid: "133abf073a614cf9bfc3c2b6fa986166" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 33543 } } peers { permanent_uuid: "a5521494881948f68156f1c53716012f" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 38427 } }
I20260812 02:32:26.383483 23744 leader_election.cc:288] T d0fac415b6bf4b1284bb7afcacf6badb P a5521494881948f68156f1c53716012f [CANDIDATE]: Term 1 election: Requested vote from peers 7a91f9c30fb24bc2abf32b1cc6fac06f (127.14.103.1:44325), 133abf073a614cf9bfc3c2b6fa986166 (127.14.103.2:33543)
I20260812 02:32:26.384027 23479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0fac415b6bf4b1284bb7afcacf6badb" candidate_uuid: "a5521494881948f68156f1c53716012f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "133abf073a614cf9bfc3c2b6fa986166"
I20260812 02:32:26.384176 23479 raft_consensus.cc:3037] T d0fac415b6bf4b1284bb7afcacf6badb P 133abf073a614cf9bfc3c2b6fa986166 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:32:26.384233 23327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0fac415b6bf4b1284bb7afcacf6badb" candidate_uuid: "a5521494881948f68156f1c53716012f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a91f9c30fb24bc2abf32b1cc6fac06f"
I20260812 02:32:26.384392 23327 raft_consensus.cc:3037] T d0fac415b6bf4b1284bb7afcacf6badb P 7a91f9c30fb24bc2abf32b1cc6fac06f [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:32:26.385099 23479 raft_consensus.cc:2454] T d0fac415b6bf4b1284bb7afcacf6badb P 133abf073a614cf9bfc3c2b6fa986166 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5521494881948f68156f1c53716012f in term 1.
I20260812 02:32:26.385408 23569 leader_election.cc:302] T d0fac415b6bf4b1284bb7afcacf6badb P a5521494881948f68156f1c53716012f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 133abf073a614cf9bfc3c2b6fa986166, a5521494881948f68156f1c53716012f; no voters:
I20260812 02:32:26.385620 23744 raft_consensus.cc:2781] T d0fac415b6bf4b1284bb7afcacf6badb P a5521494881948f68156f1c53716012f [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:32:26.385728 23327 raft_consensus.cc:2454] T d0fac415b6bf4b1284bb7afcacf6badb P 7a91f9c30fb24bc2abf32b1cc6fac06f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5521494881948f68156f1c53716012f in term 1.
I20260812 02:32:26.385762 23744 raft_consensus.cc:686] T d0fac415b6bf4b1284bb7afcacf6badb P a5521494881948f68156f1c53716012f [term 1 LEADER]: Becoming Leader. State: Replica: a5521494881948f68156f1c53716012f, State: Running, Role: LEADER
I20260812 02:32:26.386251 23744 consensus_queue.cc:237] T d0fac415b6bf4b1284bb7afcacf6badb P a5521494881948f68156f1c53716012f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a91f9c30fb24bc2abf32b1cc6fac06f" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 44325 } } peers { permanent_uuid: "133abf073a614cf9bfc3c2b6fa986166" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 33543 } } peers { permanent_uuid: "a5521494881948f68156f1c53716012f" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 38427 } }
I20260812 02:32:26.388908 23146 catalog_manager.cc:5495] T d0fac415b6bf4b1284bb7afcacf6badb P a5521494881948f68156f1c53716012f reported cstate change: term changed from 0 to 1, leader changed from <none> to a5521494881948f68156f1c53716012f (127.14.103.3). New cstate: current_term: 1 leader_uuid: "a5521494881948f68156f1c53716012f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a91f9c30fb24bc2abf32b1cc6fac06f" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 44325 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "133abf073a614cf9bfc3c2b6fa986166" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 33543 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5521494881948f68156f1c53716012f" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 38427 } health_report { overall_health: HEALTHY } } }
I20260812 02:32:26.414572 23730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:26.411425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58446 (local address 127.14.103.3:38427)
0812 02:32:26.411549 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:26.411554 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:26.411569 (+ 15us) server_negotiation.cc:407] Connection header received
0812 02:32:26.411630 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:26.411633 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:26.411689 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:26.411776 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 02:32:26.412042 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:26.412563 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:26.413452 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:26.413684 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:26.413799 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:26.414069 (+ 270us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 02:32:26.414150 (+ 81us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 02:32:26.414233 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0812 02:32:26.414332 (+ 99us) server_negotiation.cc:299] Negotiation successful
0812 02:32:26.414388 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260812 02:32:26.422640 23479 raft_consensus.cc:1261] T d0fac415b6bf4b1284bb7afcacf6badb P 133abf073a614cf9bfc3c2b6fa986166 [term 1 FOLLOWER]: Refusing update from remote peer a5521494881948f68156f1c53716012f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:32:26.422772 23327 raft_consensus.cc:1261] T d0fac415b6bf4b1284bb7afcacf6badb P 7a91f9c30fb24bc2abf32b1cc6fac06f [term 1 FOLLOWER]: Refusing update from remote peer a5521494881948f68156f1c53716012f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:32:26.423166 23744 consensus_queue.cc:1035] T d0fac415b6bf4b1284bb7afcacf6badb P a5521494881948f68156f1c53716012f [LEADER]: Connected to new peer: Peer: permanent_uuid: "133abf073a614cf9bfc3c2b6fa986166" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 33543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:32:26.423466 23790 consensus_queue.cc:1035] T d0fac415b6bf4b1284bb7afcacf6badb P a5521494881948f68156f1c53716012f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a91f9c30fb24bc2abf32b1cc6fac06f" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 44325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:32:26.462071 23148 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50152:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 02:32:26.462251 23148 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50152:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 02:32:26.464437 23148 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ff23a1d724b647f786e885dc902c2d72: Sending DeleteTablet for 3 replicas of tablet d0fac415b6bf4b1284bb7afcacf6badb
I20260812 02:32:26.465086 23305 tablet_service.cc:1514] Processing DeleteTablet for tablet d0fac415b6bf4b1284bb7afcacf6badb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:32:26 UTC) from {username='oryx', principal='oryx/127.14.103.62@KRBTEST.COM'} at 127.0.0.1:60422
I20260812 02:32:26.465175 23460 tablet_service.cc:1514] Processing DeleteTablet for tablet d0fac415b6bf4b1284bb7afcacf6badb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:32:26 UTC) from {username='oryx', principal='oryx/127.14.103.62@KRBTEST.COM'} at 127.0.0.1:56914
I20260812 02:32:26.465292 23791 tablet_replica.cc:330] T d0fac415b6bf4b1284bb7afcacf6badb P 7a91f9c30fb24bc2abf32b1cc6fac06f: stopping tablet replica
I20260812 02:32:26.465292 23624 tablet_service.cc:1514] Processing DeleteTablet for tablet d0fac415b6bf4b1284bb7afcacf6badb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:32:26 UTC) from {username='oryx', principal='oryx/127.14.103.62@KRBTEST.COM'} at 127.0.0.1:58418
I20260812 02:32:26.465381 23792 tablet_replica.cc:330] T d0fac415b6bf4b1284bb7afcacf6badb P 133abf073a614cf9bfc3c2b6fa986166: stopping tablet replica
I20260812 02:32:26.465432 23791 raft_consensus.cc:2229] T d0fac415b6bf4b1284bb7afcacf6badb P 7a91f9c30fb24bc2abf32b1cc6fac06f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 02:32:26.465456 23793 tablet_replica.cc:330] T d0fac415b6bf4b1284bb7afcacf6badb P a5521494881948f68156f1c53716012f: stopping tablet replica
I20260812 02:32:26.465489 23792 raft_consensus.cc:2229] T d0fac415b6bf4b1284bb7afcacf6badb P 133abf073a614cf9bfc3c2b6fa986166 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 02:32:26.465564 23791 raft_consensus.cc:2258] T d0fac415b6bf4b1284bb7afcacf6badb P 7a91f9c30fb24bc2abf32b1cc6fac06f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:32:26.465570 23793 raft_consensus.cc:2229] T d0fac415b6bf4b1284bb7afcacf6badb P a5521494881948f68156f1c53716012f [term 1 LEADER]: Raft consensus shutting down.
I20260812 02:32:26.465600 23792 raft_consensus.cc:2258] T d0fac415b6bf4b1284bb7afcacf6badb P 133abf073a614cf9bfc3c2b6fa986166 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:32:26.465770 23793 raft_consensus.cc:2258] T d0fac415b6bf4b1284bb7afcacf6badb P a5521494881948f68156f1c53716012f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:32:26.466593 23791 ts_tablet_manager.cc:1905] T d0fac415b6bf4b1284bb7afcacf6badb P 7a91f9c30fb24bc2abf32b1cc6fac06f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:32:26.466631 23793 ts_tablet_manager.cc:1905] T d0fac415b6bf4b1284bb7afcacf6badb P a5521494881948f68156f1c53716012f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:32:26.466939 14748 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbep4M4/build/debug/bin/kudu with pid 23233
I20260812 02:32:26.468133 23792 ts_tablet_manager.cc:1905] T d0fac415b6bf4b1284bb7afcacf6badb P 133abf073a614cf9bfc3c2b6fa986166: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:32:26.469472 23793 ts_tablet_manager.cc:1918] T d0fac415b6bf4b1284bb7afcacf6badb P a5521494881948f68156f1c53716012f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 02:32:26.469554 23793 log.cc:1196] T d0fac415b6bf4b1284bb7afcacf6badb P a5521494881948f68156f1c53716012f: Deleting WAL directory at /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501898713699-14748-0/minicluster-data/ts-2/wal/wals/d0fac415b6bf4b1284bb7afcacf6badb
I20260812 02:32:26.469827 23793 ts_tablet_manager.cc:1939] T d0fac415b6bf4b1284bb7afcacf6badb P a5521494881948f68156f1c53716012f: Deleting consensus metadata
I20260812 02:32:26.471026 23792 ts_tablet_manager.cc:1918] T d0fac415b6bf4b1284bb7afcacf6badb P 133abf073a614cf9bfc3c2b6fa986166: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 02:32:26.471109 23792 log.cc:1196] T d0fac415b6bf4b1284bb7afcacf6badb P 133abf073a614cf9bfc3c2b6fa986166: Deleting WAL directory at /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786501898713699-14748-0/minicluster-data/ts-1/wal/wals/d0fac415b6bf4b1284bb7afcacf6badb
I20260812 02:32:26.471378 23792 ts_tablet_manager.cc:1939] T d0fac415b6bf4b1284bb7afcacf6badb P 133abf073a614cf9bfc3c2b6fa986166: Deleting consensus metadata
I20260812 02:32:26.471395 23129 catalog_manager.cc:4841] TS a5521494881948f68156f1c53716012f (127.14.103.3:38427): tablet d0fac415b6bf4b1284bb7afcacf6badb (table test-table [id=d0256955a73c4d4cb6d6dc0fa6ce3dc5]) successfully deleted
I20260812 02:32:26.472286 23131 catalog_manager.cc:4841] TS 133abf073a614cf9bfc3c2b6fa986166 (127.14.103.2:33543): tablet d0fac415b6bf4b1284bb7afcacf6badb (table test-table [id=d0256955a73c4d4cb6d6dc0fa6ce3dc5]) successfully deleted
W20260812 02:32:26.476861 23130 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.103.1:44325 (error 108)
I20260812 02:32:26.477044 14748 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbep4M4/build/debug/bin/kudu with pid 23391
I20260812 02:32:26.483321 23830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:26.482726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.1:44325 (local address 127.0.0.1:60444)
0812 02:32:26.483131 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:26.483214 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.1:44325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":108,"threads_started":1}
W20260812 02:32:26.483554 23130 catalog_manager.cc:4584] TS 7a91f9c30fb24bc2abf32b1cc6fac06f (127.14.103.1:44325): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d0fac415b6bf4b1284bb7afcacf6badb: Network error: Client connection negotiation failed: client connection to 127.14.103.1:44325: connect: Connection refused (error 111)
I20260812 02:32:26.487622 14748 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbep4M4/build/debug/bin/kudu with pid 23549
I20260812 02:32:26.499028 14748 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbep4M4/build/debug/bin/kudu with pid 23111
2026-08-12T02:32:26Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3685 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786501898713699-14748-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 02:32:26 dist-test-slave-2w3w krb5kdc[23835](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 02:32:26 dist-test-slave-2w3w krb5kdc[23835](info): set up 2 sockets
Aug 12 02:32:26 dist-test-slave-2w3w krb5kdc[23835](info): commencing operation
krb5kdc: starting...
W20260812 02:32:27.653978 14748 mini_kdc.cc:121] Time spent starting KDC: real 1.125s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786501898713699-14748-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786501898713699-14748-0/krb5kdc/test-user.keytab.
Aug 12 02:32:27 dist-test-slave-2w3w krb5kdc[23835](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501947, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T02:32:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T02:32:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.103.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.103.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.103.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.62.keytab.
Entry for principal kudu/127.14.103.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.62.keytab.
I20260812 02:32:27.811416 14748 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbep4M4/build/debug/bin/kudu
/tmp/dist-test-taskbep4M4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786501898713699-14748-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786501898713699-14748-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786501898713699-14748-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786501898713699-14748-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.103.62:33395
--webserver_interface=127.14.103.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.62.keytab
--principal=kudu/127.14.103.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.103.20:35665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.103.62:33395
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786501898713699-14748-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786501898713699-14748-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786501898713699-14748-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:32:27.943440 23851 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:32:27.943892 23851 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:32:27.943992 23851 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:32:27.948408 23851 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:32:27.948518 23851 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:32:27.948541 23851 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:32:27.948781 23851 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:32:27.948853 23851 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 02:32:27.956837 23851 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.103.20:35665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786501898713699-14748-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786501898713699-14748-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.103.62:33395
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.103.62:33395
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786501898713699-14748-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.103.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23851
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786501898713699-14748-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:32:27.958184 23851 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:32:27.959575 23851 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:32:27.969380 23860 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:32:27.969305 23857 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:32:27.969305 23858 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:32:27.969861 23851 server_base.cc:1034] running on GCE node
I20260812 02:32:27.970690 23851 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:32:27.972136 23851 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:32:27.973407 23851 hybrid_clock.cc:648] HybridClock initialized: now 1786501947973337 us; error 96 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-08-12T02:32:27Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1487 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501898713699-14748-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 02:32:28 dist-test-slave-2w3w krb5kdc[23865](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 02:32:28 dist-test-slave-2w3w krb5kdc[23865](info): set up 2 sockets
Aug 12 02:32:28 dist-test-slave-2w3w krb5kdc[23865](info): commencing operation
krb5kdc: starting...
W20260812 02:32:29.170737 14748 mini_kdc.cc:121] Time spent starting KDC: real 1.155s user 0.005s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501898713699-14748-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501898713699-14748-0/krb5kdc/test-user.keytab.
Aug 12 02:32:29 dist-test-slave-2w3w krb5kdc[23865](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501949, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T02:32:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T02:32:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.103.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.103.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.103.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.62.keytab.
Entry for principal kudu/127.14.103.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.62.keytab.
I20260812 02:32:29.328150 14748 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbep4M4/build/debug/bin/kudu
/tmp/dist-test-taskbep4M4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501898713699-14748-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501898713699-14748-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501898713699-14748-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501898713699-14748-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.103.62:44127
--webserver_interface=127.14.103.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.62.keytab
--principal=kudu/127.14.103.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.103.20:42525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.103.62:44127
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501898713699-14748-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501898713699-14748-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501898713699-14748-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:32:29.458664 23880 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:32:29.459039 23880 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:32:29.459088 23880 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:32:29.463671 23880 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:32:29.463758 23880 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:32:29.463779 23880 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:32:29.463797 23880 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:32:29.463812 23880 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 02:32:29.470253 23880 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.103.20:42525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501898713699-14748-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501898713699-14748-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.103.62:44127
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.62.keytab
--principal=kudu/127.14.103.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.103.62:44127
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501898713699-14748-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.103.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23880
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501898713699-14748-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:32:29.471550 23880 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:32:29.472723 23880 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:32:29.482756 23888 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:32:29.482951 23886 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:32:29.482828 23885 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:32:29.482930 23880 server_base.cc:1034] running on GCE node
I20260812 02:32:29.484025 23880 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:32:29.485272 23880 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:32:29.486470 23880 hybrid_clock.cc:648] HybridClock initialized: now 1786501949486451 us; error 85 us; skew 500 ppm
Aug 12 02:32:29 dist-test-slave-2w3w krb5kdc[23865](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501949, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:32:29.489738 23880 init.cc:377] Logged in from keytab as kudu/127.14.103.62@KRBTEST.COM (short username kudu)
I20260812 02:32:29.490849 23880 webserver.cc:460] Webserver started at http://127.14.103.62:38317/ using document root <none> and password file <none>
I20260812 02:32:29.491518 23880 fs_manager.cc:359] Metadata directory not provided
I20260812 02:32:29.491622 23880 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:32:29.491871 23880 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:32:29.494167 23880 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501898713699-14748-0/minicluster-data/master-0/data/instance:
uuid: "9d1cf575f9e24f5c8b91051fc00f95a0"
format_stamp: "Formatted at 2026-08-12 02:32:29 on dist-test-slave-2w3w"
server_key: "4a9a6f9c957347e3029d14f9ae35072c"
server_key_iv: "9a057ec6fafbb916780fa602a38795f1"
server_key_version: "encryptionkey@0"
I20260812 02:32:29.494844 23880 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501898713699-14748-0/minicluster-data/master-0/wal/instance:
uuid: "9d1cf575f9e24f5c8b91051fc00f95a0"
format_stamp: "Formatted at 2026-08-12 02:32:29 on dist-test-slave-2w3w"
server_key: "4a9a6f9c957347e3029d14f9ae35072c"
server_key_iv: "9a057ec6fafbb916780fa602a38795f1"
server_key_version: "encryptionkey@0"
I20260812 02:32:29.499118 23880 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260812 02:32:29.502013 23896 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:32:29.503633 23880 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 02:32:29.503813 23880 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501898713699-14748-0/minicluster-data/master-0/data,/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501898713699-14748-0/minicluster-data/master-0/wal
uuid: "9d1cf575f9e24f5c8b91051fc00f95a0"
format_stamp: "Formatted at 2026-08-12 02:32:29 on dist-test-slave-2w3w"
server_key: "4a9a6f9c957347e3029d14f9ae35072c"
server_key_iv: "9a057ec6fafbb916780fa602a38795f1"
server_key_version: "encryptionkey@0"
I20260812 02:32:29.503942 23880 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501898713699-14748-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501898713699-14748-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501898713699-14748-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:32:29.519886 23880 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:32:29.523578 23880 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:32:29.523844 23880 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:32:29.532187 23880 rpc_server.cc:307] RPC server started. Bound to: 127.14.103.62:44127
I20260812 02:32:29.532214 23954 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.103.62:44127 every 8 connection(s)
I20260812 02:32:29.533274 23880 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501898713699-14748-0/minicluster-data/master-0/data/info.pb
I20260812 02:32:29.536379 23955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:32:29.537446 14748 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbep4M4/build/debug/bin/kudu as pid 23880
I20260812 02:32:29.537611 14748 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501898713699-14748-0/minicluster-data/master-0/wal/instance
I20260812 02:32:29.537914 14748 external_mini_cluster.cc:1428] Setting key 60b045b6bf596dc928b73ed3841f2d06
I20260812 02:32:29.543890 23955 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d1cf575f9e24f5c8b91051fc00f95a0: Bootstrap starting.
I20260812 02:32:29.546890 23955 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9d1cf575f9e24f5c8b91051fc00f95a0: Neither blocks nor log segments found. Creating new log.
I20260812 02:32:29.547899 23955 log.cc:824] T 00000000000000000000000000000000 P 9d1cf575f9e24f5c8b91051fc00f95a0: Log is configured to *not* fsync() on all Append() calls
Aug 12 02:32:29 dist-test-slave-2w3w krb5kdc[23865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501949, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.103.62@KRBTEST.COM
I20260812 02:32:29.550673 23955 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d1cf575f9e24f5c8b91051fc00f95a0: No bootstrap required, opened a new log
I20260812 02:32:29.556183 23958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:29.539567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34684 (local address 127.14.103.62:44127)
0812 02:32:29.540079 (+ 512us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:29.540099 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:29.540147 (+ 48us) server_negotiation.cc:407] Connection header received
0812 02:32:29.541057 (+ 910us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:29.541083 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:29.541563 (+ 480us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:29.541987 (+ 424us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:32:29.542943 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:29.544461 (+ 1518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:29.545474 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:29.546182 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:29.546357 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:29.549555 (+ 3198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:32:29.549562 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:32:29.549581 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:32:29.549616 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:32:29.551954 (+ 2338us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:29.552545 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:29.552552 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:29.552562 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:29.552737 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:29.553058 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:29.553063 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:29.553068 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:29.553803 (+ 735us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:32:29.554023 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0812 02:32:29.554598 (+ 575us) server_negotiation.cc:299] Negotiation successful
0812 02:32:29.555054 (+ 456us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":154,"threads_started":1}
I20260812 02:32:29.556000 23955 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9d1cf575f9e24f5c8b91051fc00f95a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d1cf575f9e24f5c8b91051fc00f95a0" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 44127 } }
I20260812 02:32:29.556304 23955 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9d1cf575f9e24f5c8b91051fc00f95a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:32:29.556350 23955 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9d1cf575f9e24f5c8b91051fc00f95a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d1cf575f9e24f5c8b91051fc00f95a0, State: Initialized, Role: FOLLOWER
I20260812 02:32:29.556835 23955 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9d1cf575f9e24f5c8b91051fc00f95a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d1cf575f9e24f5c8b91051fc00f95a0" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 44127 } }
I20260812 02:32:29.557017 23955 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9d1cf575f9e24f5c8b91051fc00f95a0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 02:32:29.557092 23955 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9d1cf575f9e24f5c8b91051fc00f95a0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 02:32:29.557200 23955 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9d1cf575f9e24f5c8b91051fc00f95a0 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:32:29.558558 23955 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9d1cf575f9e24f5c8b91051fc00f95a0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d1cf575f9e24f5c8b91051fc00f95a0" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 44127 } }
I20260812 02:32:29.558951 23955 leader_election.cc:302] T 00000000000000000000000000000000 P 9d1cf575f9e24f5c8b91051fc00f95a0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9d1cf575f9e24f5c8b91051fc00f95a0; no voters:
I20260812 02:32:29.559396 23955 leader_election.cc:288] T 00000000000000000000000000000000 P 9d1cf575f9e24f5c8b91051fc00f95a0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 02:32:29.559572 23960 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9d1cf575f9e24f5c8b91051fc00f95a0 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:32:29.559883 23960 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9d1cf575f9e24f5c8b91051fc00f95a0 [term 1 LEADER]: Becoming Leader. State: Replica: 9d1cf575f9e24f5c8b91051fc00f95a0, State: Running, Role: LEADER
I20260812 02:32:29.560300 23960 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9d1cf575f9e24f5c8b91051fc00f95a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d1cf575f9e24f5c8b91051fc00f95a0" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 44127 } }
I20260812 02:32:29.561007 23955 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9d1cf575f9e24f5c8b91051fc00f95a0 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:32:29.562216 23962 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d1cf575f9e24f5c8b91051fc00f95a0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9d1cf575f9e24f5c8b91051fc00f95a0. Latest consensus state: current_term: 1 leader_uuid: "9d1cf575f9e24f5c8b91051fc00f95a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d1cf575f9e24f5c8b91051fc00f95a0" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 44127 } } }
I20260812 02:32:29.562227 23961 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d1cf575f9e24f5c8b91051fc00f95a0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9d1cf575f9e24f5c8b91051fc00f95a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d1cf575f9e24f5c8b91051fc00f95a0" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 44127 } } }
I20260812 02:32:29.562378 23961 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d1cf575f9e24f5c8b91051fc00f95a0 [sys.catalog]: This master's current role is: LEADER
I20260812 02:32:29.562378 23962 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d1cf575f9e24f5c8b91051fc00f95a0 [sys.catalog]: This master's current role is: LEADER
I20260812 02:32:29.563159 23970 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 02:32:29.566642 23970 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 02:32:29.572474 23970 catalog_manager.cc:1347] Generated new cluster ID: 5c79fdec80b2439eb3020cb7d9d58bb2
I20260812 02:32:29.572594 23970 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 02:32:29.591522 23970 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 02:32:29.593024 23970 catalog_manager.cc:1504] Loading token signing keys...
I20260812 02:32:29.605985 23970 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9d1cf575f9e24f5c8b91051fc00f95a0: Generated new TSK 0
I20260812 02:32:29.606823 23970 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.103.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.103.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.103.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.1.keytab.
Entry for principal kudu/127.14.103.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.1.keytab.
I20260812 02:32:29.650710 14748 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbep4M4/build/debug/bin/kudu
/tmp/dist-test-taskbep4M4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501898713699-14748-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501898713699-14748-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501898713699-14748-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501898713699-14748-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.103.1:0
--local_ip_for_outbound_sockets=127.14.103.1
--webserver_interface=127.14.103.1
--webserver_port=0
--tserver_master_addrs=127.14.103.62:44127
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.1.keytab
--principal=kudu/127.14.103.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.103.20:42525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501898713699-14748-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501898713699-14748-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501898713699-14748-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:32:29.776981 23982 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:32:29.777277 23982 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:32:29.777324 23982 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:32:29.781658 23982 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:32:29.781754 23982 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:32:29.781819 23982 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.103.1
I20260812 02:32:29.787458 23982 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.103.20:42525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501898713699-14748-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501898713699-14748-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.1.keytab
--principal=kudu/127.14.103.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.103.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501898713699-14748-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.103.1
--webserver_port=0
--tserver_master_addrs=127.14.103.62:44127
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23982
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.103.1
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501898713699-14748-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:32:29.788677 23982 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:32:29.789753 23982 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:32:29.801561 23988 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:32:29.801491 23990 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:32:29.801491 23992 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:32:29.802034 23982 server_base.cc:1034] running on GCE node
I20260812 02:32:29.802620 23982 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:32:29.803416 23982 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:32:29.804581 23982 hybrid_clock.cc:648] HybridClock initialized: now 1786501949804561 us; error 45 us; skew 500 ppm
Aug 12 02:32:29 dist-test-slave-2w3w krb5kdc[23865](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501949, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:32:29.808197 23982 init.cc:377] Logged in from keytab as kudu/127.14.103.1@KRBTEST.COM (short username kudu)
I20260812 02:32:29.809327 23982 webserver.cc:460] Webserver started at http://127.14.103.1:40803/ using document root <none> and password file <none>
I20260812 02:32:29.809963 23982 fs_manager.cc:359] Metadata directory not provided
I20260812 02:32:29.810067 23982 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:32:29.810325 23982 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:32:29.812652 23982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501898713699-14748-0/minicluster-data/ts-0/data/instance:
uuid: "c0bd44166b62428fa1941d3589cb060c"
format_stamp: "Formatted at 2026-08-12 02:32:29 on dist-test-slave-2w3w"
server_key: "43ac2b9b2ccdfef3f20298e6be7c2dec"
server_key_iv: "c51f4d450ac57f0f433bab6953d810ee"
server_key_version: "encryptionkey@0"
I20260812 02:32:29.813313 23982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501898713699-14748-0/minicluster-data/ts-0/wal/instance:
uuid: "c0bd44166b62428fa1941d3589cb060c"
format_stamp: "Formatted at 2026-08-12 02:32:29 on dist-test-slave-2w3w"
server_key: "43ac2b9b2ccdfef3f20298e6be7c2dec"
server_key_iv: "c51f4d450ac57f0f433bab6953d810ee"
server_key_version: "encryptionkey@0"
I20260812 02:32:29.817668 23982 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260812 02:32:29.820591 24000 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:32:29.822118 23982 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260812 02:32:29.822299 23982 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501898713699-14748-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501898713699-14748-0/minicluster-data/ts-0/wal
uuid: "c0bd44166b62428fa1941d3589cb060c"
format_stamp: "Formatted at 2026-08-12 02:32:29 on dist-test-slave-2w3w"
server_key: "43ac2b9b2ccdfef3f20298e6be7c2dec"
server_key_iv: "c51f4d450ac57f0f433bab6953d810ee"
server_key_version: "encryptionkey@0"
I20260812 02:32:29.822446 23982 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501898713699-14748-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501898713699-14748-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501898713699-14748-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:32:29.846101 23982 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:32:29.850103 23982 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:32:29.850347 23982 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:32:29.851042 23982 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:32:29.851967 23982 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:32:29.852044 23982 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:32:29.852133 23982 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:32:29.852190 23982 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:32:29.862540 23982 rpc_server.cc:307] RPC server started. Bound to: 127.14.103.1:43433
I20260812 02:32:29.862560 24129 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.103.1:43433 every 8 connection(s)
I20260812 02:32:29.863970 23982 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501898713699-14748-0/minicluster-data/ts-0/data/info.pb
I20260812 02:32:29.869645 14748 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbep4M4/build/debug/bin/kudu as pid 23982
I20260812 02:32:29.869803 14748 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501898713699-14748-0/minicluster-data/ts-0/wal/instance
I20260812 02:32:29.870136 14748 external_mini_cluster.cc:1428] Setting key 698601b106e7d4d9d828b2cc945607c6
Aug 12 02:32:29 dist-test-slave-2w3w krb5kdc[23865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501949, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.1@KRBTEST.COM for kudu/127.14.103.62@KRBTEST.COM
I20260812 02:32:29.882164 23958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:29.866198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.1:49327 (local address 127.14.103.62:44127)
0812 02:32:29.866356 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:29.866360 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:29.867618 (+ 1258us) server_negotiation.cc:407] Connection header received
0812 02:32:29.868590 (+ 972us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:29.868594 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:29.868655 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:29.868770 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:32:29.870484 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:29.871272 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:29.873120 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:29.873383 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:29.873540 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:29.877057 (+ 3517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:32:29.877064 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:32:29.877070 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:32:29.877110 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:32:29.879270 (+ 2160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:29.879952 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:29.879957 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:29.879959 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:29.880068 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:29.880497 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:29.880503 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:29.880507 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:29.880808 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:32:29.880956 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0812 02:32:29.881764 (+ 808us) server_negotiation.cc:299] Negotiation successful
0812 02:32:29.881945 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
WARNING: no policy specified for kudu/127.14.103.2@KRBTEST.COM; defaulting to no policy
I20260812 02:32:29.883210 24132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:29.866537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:44127 (local address 127.14.103.1:49327)
0812 02:32:29.867440 (+ 903us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:29.867490 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0812 02:32:29.868363 (+ 873us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:32:29.868963 (+ 600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:32:29.868971 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:32:29.869500 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:32:29.870269 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:29.870294 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:29.871437 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:29.871441 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:29.872943 (+ 1502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:29.872962 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:29.873611 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:29.873618 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:29.873895 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:29.874737 (+ 842us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:32:29.874767 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:32:29.876800 (+ 2033us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:32:29.879453 (+ 2653us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:32:29.879460 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:32:29.879486 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:32:29.879824 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:32:29.880208 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:32:29.880210 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:32:29.880212 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:32:29.880382 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:32:29.880974 (+ 592us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:32:29.880980 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:32:29.881469 (+ 489us) client_negotiation.cc:769] Sending connection context
0812 02:32:29.881663 (+ 194us) client_negotiation.cc:240] Negotiation successful
0812 02:32:29.882015 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":689,"thread_start_us":143,"threads_started":1}
I20260812 02:32:29.884770 24130 heartbeater.cc:347] Connected to a master server at 127.14.103.62:44127
I20260812 02:32:29.885118 24130 heartbeater.cc:464] Registering TS with master...
I20260812 02:32:29.885807 24130 heartbeater.cc:511] Master 127.14.103.62:44127 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.103.2@KRBTEST.COM" created.
I20260812 02:32:29.887712 23916 ts_manager.cc:194] Registered new tserver with Master: c0bd44166b62428fa1941d3589cb060c (127.14.103.1:43433)
I20260812 02:32:29.888931 23916 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.103.1@KRBTEST.COM'} at 127.14.103.1:49327
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.103.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.2.keytab.
Entry for principal kudu/127.14.103.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.2.keytab.
I20260812 02:32:29.907259 14748 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbep4M4/build/debug/bin/kudu
/tmp/dist-test-taskbep4M4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501898713699-14748-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501898713699-14748-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501898713699-14748-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501898713699-14748-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.103.2:0
--local_ip_for_outbound_sockets=127.14.103.2
--webserver_interface=127.14.103.2
--webserver_port=0
--tserver_master_addrs=127.14.103.62:44127
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.2.keytab
--principal=kudu/127.14.103.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.103.20:42525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501898713699-14748-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501898713699-14748-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501898713699-14748-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:32:30.030988 24136 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:32:30.031316 24136 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:32:30.031364 24136 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:32:30.035851 24136 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:32:30.035929 24136 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:32:30.035995 24136 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.103.2
I20260812 02:32:30.043219 24136 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.103.20:42525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501898713699-14748-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501898713699-14748-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.2.keytab
--principal=kudu/127.14.103.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.103.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501898713699-14748-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.103.2
--webserver_port=0
--tserver_master_addrs=127.14.103.62:44127
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24136
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.103.2
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501898713699-14748-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:32:30.044390 24136 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:32:30.045559 24136 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:32:30.055801 24141 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:32:30.055830 24147 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:32:30.056187 24136 server_base.cc:1034] running on GCE node
W20260812 02:32:30.055805 24145 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:32:30.056720 24136 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:32:30.057448 24136 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:32:30.058673 24136 hybrid_clock.cc:648] HybridClock initialized: now 1786501950058658 us; error 30 us; skew 500 ppm
Aug 12 02:32:30 dist-test-slave-2w3w krb5kdc[23865](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501950, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:32:30.062112 24136 init.cc:377] Logged in from keytab as kudu/127.14.103.2@KRBTEST.COM (short username kudu)
I20260812 02:32:30.063299 24136 webserver.cc:460] Webserver started at http://127.14.103.2:34421/ using document root <none> and password file <none>
I20260812 02:32:30.064100 24136 fs_manager.cc:359] Metadata directory not provided
I20260812 02:32:30.064213 24136 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:32:30.064476 24136 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:32:30.066672 24136 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501898713699-14748-0/minicluster-data/ts-1/data/instance:
uuid: "3a778b3f58274d0ab008fdf38dc5ad9b"
format_stamp: "Formatted at 2026-08-12 02:32:30 on dist-test-slave-2w3w"
server_key: "f73de3436ea4c013d2fa257d369f4c76"
server_key_iv: "ecfca5d0e8a265d86e21118ed27f156c"
server_key_version: "encryptionkey@0"
I20260812 02:32:30.067328 24136 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501898713699-14748-0/minicluster-data/ts-1/wal/instance:
uuid: "3a778b3f58274d0ab008fdf38dc5ad9b"
format_stamp: "Formatted at 2026-08-12 02:32:30 on dist-test-slave-2w3w"
server_key: "f73de3436ea4c013d2fa257d369f4c76"
server_key_iv: "ecfca5d0e8a265d86e21118ed27f156c"
server_key_version: "encryptionkey@0"
I20260812 02:32:30.071523 24136 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260812 02:32:30.074230 24155 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:32:30.075677 24136 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 02:32:30.075841 24136 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501898713699-14748-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501898713699-14748-0/minicluster-data/ts-1/wal
uuid: "3a778b3f58274d0ab008fdf38dc5ad9b"
format_stamp: "Formatted at 2026-08-12 02:32:30 on dist-test-slave-2w3w"
server_key: "f73de3436ea4c013d2fa257d369f4c76"
server_key_iv: "ecfca5d0e8a265d86e21118ed27f156c"
server_key_version: "encryptionkey@0"
I20260812 02:32:30.075986 24136 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501898713699-14748-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501898713699-14748-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501898713699-14748-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:32:30.113811 24136 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:32:30.117686 24136 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:32:30.117939 24136 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:32:30.118623 24136 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:32:30.119628 24136 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:32:30.119716 24136 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:32:30.119814 24136 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:32:30.119866 24136 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:32:30.130292 24136 rpc_server.cc:307] RPC server started. Bound to: 127.14.103.2:34345
I20260812 02:32:30.130313 24281 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.103.2:34345 every 8 connection(s)
I20260812 02:32:30.131397 24136 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501898713699-14748-0/minicluster-data/ts-1/data/info.pb
I20260812 02:32:30.136552 14748 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbep4M4/build/debug/bin/kudu as pid 24136
I20260812 02:32:30.136672 14748 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501898713699-14748-0/minicluster-data/ts-1/wal/instance
I20260812 02:32:30.136984 14748 external_mini_cluster.cc:1428] Setting key dd17c969448eea39f8d00f571cb5665c
Aug 12 02:32:30 dist-test-slave-2w3w krb5kdc[23865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501950, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.2@KRBTEST.COM for kudu/127.14.103.62@KRBTEST.COM
I20260812 02:32:30.149369 23958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:30.133471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.2:45485 (local address 127.14.103.62:44127)
0812 02:32:30.133615 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:30.133618 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:30.134823 (+ 1205us) server_negotiation.cc:407] Connection header received
0812 02:32:30.135930 (+ 1107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:30.135934 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:30.136000 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:30.136080 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:32:30.137766 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:30.138313 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:30.140236 (+ 1923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:30.140498 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:30.140603 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:30.144112 (+ 3509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:32:30.144117 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:32:30.144120 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:32:30.144161 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:32:30.146224 (+ 2063us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:30.146949 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:30.146954 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:30.146989 (+ 35us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:30.147109 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:30.147718 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:30.147723 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:30.147725 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:30.148035 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:32:30.148171 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0812 02:32:30.148841 (+ 670us) server_negotiation.cc:299] Negotiation successful
0812 02:32:30.149007 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
WARNING: no policy specified for kudu/127.14.103.3@KRBTEST.COM; defaulting to no policy
I20260812 02:32:30.150107 24284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:30.133819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:44127 (local address 127.14.103.2:45485)
0812 02:32:30.134598 (+ 779us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:30.134649 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0812 02:32:30.135678 (+ 1029us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:32:30.136377 (+ 699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:32:30.136386 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:32:30.136905 (+ 519us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:32:30.137595 (+ 690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:30.137627 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:30.138486 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:30.138489 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:30.140046 (+ 1557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:30.140064 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:30.140703 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:30.140711 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:30.140972 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:30.141857 (+ 885us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:32:30.141875 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:32:30.143850 (+ 1975us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:32:30.146423 (+ 2573us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:32:30.146431 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:32:30.146449 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:32:30.146802 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:32:30.147256 (+ 454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:32:30.147258 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:32:30.147260 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:32:30.147465 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:32:30.148231 (+ 766us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:32:30.148237 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:32:30.148539 (+ 302us) client_negotiation.cc:769] Sending connection context
0812 02:32:30.148715 (+ 176us) client_negotiation.cc:240] Negotiation successful
0812 02:32:30.149026 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":562,"thread_start_us":134,"threads_started":1}
I20260812 02:32:30.151810 24282 heartbeater.cc:347] Connected to a master server at 127.14.103.62:44127
I20260812 02:32:30.152124 24282 heartbeater.cc:464] Registering TS with master...
I20260812 02:32:30.152819 24282 heartbeater.cc:511] Master 127.14.103.62:44127 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.103.3@KRBTEST.COM" created.
I20260812 02:32:30.154241 23916 ts_manager.cc:194] Registered new tserver with Master: 3a778b3f58274d0ab008fdf38dc5ad9b (127.14.103.2:34345)
I20260812 02:32:30.154933 23916 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.103.2@KRBTEST.COM'} at 127.14.103.2:45485
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.103.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.3.keytab.
Entry for principal kudu/127.14.103.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.3.keytab.
I20260812 02:32:30.175369 14748 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbep4M4/build/debug/bin/kudu
/tmp/dist-test-taskbep4M4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501898713699-14748-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501898713699-14748-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501898713699-14748-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501898713699-14748-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.103.3:0
--local_ip_for_outbound_sockets=127.14.103.3
--webserver_interface=127.14.103.3
--webserver_port=0
--tserver_master_addrs=127.14.103.62:44127
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.3.keytab
--principal=kudu/127.14.103.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.103.20:42525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501898713699-14748-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501898713699-14748-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501898713699-14748-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:32:30.305022 24287 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:32:30.305346 24287 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:32:30.305395 24287 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:32:30.309994 24287 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:32:30.310091 24287 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:32:30.310158 24287 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.103.3
I20260812 02:32:30.316187 24287 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.103.20:42525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501898713699-14748-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501898713699-14748-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.3.keytab
--principal=kudu/127.14.103.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.103.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501898713699-14748-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.103.3
--webserver_port=0
--tserver_master_addrs=127.14.103.62:44127
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24287
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.103.3
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501898713699-14748-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:32:30.317516 24287 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:32:30.318635 24287 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:32:30.330608 24294 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:32:30.330556 24295 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:32:30.330832 24287 server_base.cc:1034] running on GCE node
W20260812 02:32:30.330556 24297 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:32:30.331486 24287 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:32:30.332366 24287 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:32:30.333559 24287 hybrid_clock.cc:648] HybridClock initialized: now 1786501950333544 us; error 33 us; skew 500 ppm
Aug 12 02:32:30 dist-test-slave-2w3w krb5kdc[23865](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501950, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:32:30.336730 24287 init.cc:377] Logged in from keytab as kudu/127.14.103.3@KRBTEST.COM (short username kudu)
I20260812 02:32:30.337805 24287 webserver.cc:460] Webserver started at http://127.14.103.3:35455/ using document root <none> and password file <none>
I20260812 02:32:30.338440 24287 fs_manager.cc:359] Metadata directory not provided
I20260812 02:32:30.338542 24287 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:32:30.338804 24287 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:32:30.341142 24287 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501898713699-14748-0/minicluster-data/ts-2/data/instance:
uuid: "478dcc6d5fc242399cdff84b438b2919"
format_stamp: "Formatted at 2026-08-12 02:32:30 on dist-test-slave-2w3w"
server_key: "2554373c7eff74890547e80d20176c52"
server_key_iv: "ad3c4940c277e9f8219a74c9148f1a3b"
server_key_version: "encryptionkey@0"
I20260812 02:32:30.341791 24287 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501898713699-14748-0/minicluster-data/ts-2/wal/instance:
uuid: "478dcc6d5fc242399cdff84b438b2919"
format_stamp: "Formatted at 2026-08-12 02:32:30 on dist-test-slave-2w3w"
server_key: "2554373c7eff74890547e80d20176c52"
server_key_iv: "ad3c4940c277e9f8219a74c9148f1a3b"
server_key_version: "encryptionkey@0"
I20260812 02:32:30.346030 24287 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260812 02:32:30.348879 24305 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:32:30.350265 24287 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260812 02:32:30.350447 24287 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501898713699-14748-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501898713699-14748-0/minicluster-data/ts-2/wal
uuid: "478dcc6d5fc242399cdff84b438b2919"
format_stamp: "Formatted at 2026-08-12 02:32:30 on dist-test-slave-2w3w"
server_key: "2554373c7eff74890547e80d20176c52"
server_key_iv: "ad3c4940c277e9f8219a74c9148f1a3b"
server_key_version: "encryptionkey@0"
I20260812 02:32:30.350636 24287 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501898713699-14748-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501898713699-14748-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501898713699-14748-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:32:30.373036 24287 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:32:30.376639 24287 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:32:30.376916 24287 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:32:30.377586 24287 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:32:30.378508 24287 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:32:30.378587 24287 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:32:30.378665 24287 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:32:30.378710 24287 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:32:30.389798 24287 rpc_server.cc:307] RPC server started. Bound to: 127.14.103.3:41919
I20260812 02:32:30.389840 24430 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.103.3:41919 every 8 connection(s)
I20260812 02:32:30.391055 24287 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501898713699-14748-0/minicluster-data/ts-2/data/info.pb
I20260812 02:32:30.394856 14748 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbep4M4/build/debug/bin/kudu as pid 24287
I20260812 02:32:30.395046 14748 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501898713699-14748-0/minicluster-data/ts-2/wal/instance
I20260812 02:32:30.395347 14748 external_mini_cluster.cc:1428] Setting key 0f7e1d1654d55ea32f6dc2270a3d4678
Aug 12 02:32:30 dist-test-slave-2w3w krb5kdc[23865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501950, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.3@KRBTEST.COM for kudu/127.14.103.62@KRBTEST.COM
I20260812 02:32:30.409771 23958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:30.393332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.3:54303 (local address 127.14.103.62:44127)
0812 02:32:30.393558 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:30.393563 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:30.394697 (+ 1134us) server_negotiation.cc:407] Connection header received
0812 02:32:30.396020 (+ 1323us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:30.396024 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:30.396084 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:30.396228 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:32:30.398069 (+ 1841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:30.398708 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:30.400081 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:30.400541 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:30.401086 (+ 545us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:30.404538 (+ 3452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:32:30.404541 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:32:30.404544 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:32:30.404574 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:32:30.406489 (+ 1915us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:30.407369 (+ 880us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:30.407373 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:30.407374 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:30.407466 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:30.408021 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:30.408024 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:30.408025 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:30.408255 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:32:30.408347 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0812 02:32:30.409358 (+ 1011us) server_negotiation.cc:299] Negotiation successful
0812 02:32:30.409568 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116}
I20260812 02:32:30.411036 24434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:30.393681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:44127 (local address 127.14.103.3:54303)
0812 02:32:30.394504 (+ 823us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:30.394556 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0812 02:32:30.395675 (+ 1119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:32:30.396546 (+ 871us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:32:30.396557 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:32:30.397120 (+ 563us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:32:30.397887 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:30.397911 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:30.398853 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:30.398857 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:30.399939 (+ 1082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:30.399950 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:30.401130 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:30.401134 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:30.401366 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:30.402070 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:32:30.402088 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:32:30.404110 (+ 2022us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:32:30.406659 (+ 2549us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:32:30.406667 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:32:30.406683 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:32:30.407194 (+ 511us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:32:30.407586 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:32:30.407589 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:32:30.407591 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:32:30.407818 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:32:30.408470 (+ 652us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:32:30.408481 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:32:30.408882 (+ 401us) client_negotiation.cc:769] Sending connection context
0812 02:32:30.409254 (+ 372us) client_negotiation.cc:240] Negotiation successful
0812 02:32:30.409747 (+ 493us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":574,"thread_start_us":135,"threads_started":1}
I20260812 02:32:30.412535 24432 heartbeater.cc:347] Connected to a master server at 127.14.103.62:44127
I20260812 02:32:30.412952 24432 heartbeater.cc:464] Registering TS with master...
I20260812 02:32:30.413637 24432 heartbeater.cc:511] Master 127.14.103.62:44127 requested a full tablet report, sending...
I20260812 02:32:30.415024 23916 ts_manager.cc:194] Registered new tserver with Master: 478dcc6d5fc242399cdff84b438b2919 (127.14.103.3:41919)
I20260812 02:32:30.415658 23916 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.103.3@KRBTEST.COM'} at 127.14.103.3:54303
I20260812 02:32:30.424080 14748 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.14.103.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.103.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.103.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501898713699-14748-0/krb5kdc/oryx_127.14.103.1.keytab.
Entry for principal oryx/127.14.103.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501898713699-14748-0/krb5kdc/oryx_127.14.103.1.keytab.
WARNING: no policy specified for oryx/127.14.103.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.103.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.103.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501898713699-14748-0/krb5kdc/oryx_127.14.103.2.keytab.
Entry for principal oryx/127.14.103.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501898713699-14748-0/krb5kdc/oryx_127.14.103.2.keytab.
WARNING: no policy specified for oryx/127.14.103.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.103.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.103.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501898713699-14748-0/krb5kdc/oryx_127.14.103.3.keytab.
Entry for principal oryx/127.14.103.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501898713699-14748-0/krb5kdc/oryx_127.14.103.3.keytab.
I20260812 02:32:30.528105 14748 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbep4M4/build/debug/bin/kudu with pid 23982
I20260812 02:32:30.538520 14748 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbep4M4/build/debug/bin/kudu with pid 24136
I20260812 02:32:30.546702 14748 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbep4M4/build/debug/bin/kudu with pid 24287
I20260812 02:32:30.555589 14748 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbep4M4/build/debug/bin/kudu with pid 23880
I20260812 02:32:30.565191 14748 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbep4M4/build/debug/bin/kudu
/tmp/dist-test-taskbep4M4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501898713699-14748-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501898713699-14748-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501898713699-14748-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501898713699-14748-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.103.62:44127
--webserver_interface=127.14.103.62
--webserver_port=38317
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.62.keytab
--principal=kudu/127.14.103.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.103.20:42525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.103.62:44127
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501898713699-14748-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501898713699-14748-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501898713699-14748-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:32:30.694649 24443 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:32:30.695096 24443 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:32:30.695144 24443 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:32:30.699565 24443 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:32:30.699662 24443 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:32:30.699683 24443 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:32:30.699700 24443 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:32:30.699717 24443 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 02:32:30.705592 24443 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.103.20:42525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501898713699-14748-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501898713699-14748-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.103.62:44127
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.62.keytab
--principal=kudu/127.14.103.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.103.62:44127
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501898713699-14748-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.103.62
--webserver_port=38317
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24443
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501898713699-14748-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:32:30.706835 24443 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:32:30.707966 24443 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:32:30.718138 24451 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:32:30.718178 24450 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:32:30.718178 24454 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:32:30.718629 24443 server_base.cc:1034] running on GCE node
I20260812 02:32:30.719509 24443 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:32:30.720835 24443 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:32:30.722082 24443 hybrid_clock.cc:648] HybridClock initialized: now 1786501950722045 us; error 54 us; skew 500 ppm
Aug 12 02:32:30 dist-test-slave-2w3w krb5kdc[23865](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501950, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:32:30.725519 24443 init.cc:377] Logged in from keytab as kudu/127.14.103.62@KRBTEST.COM (short username kudu)
I20260812 02:32:30.726583 24443 webserver.cc:460] Webserver started at http://127.14.103.62:38317/ using document root <none> and password file <none>
I20260812 02:32:30.727279 24443 fs_manager.cc:359] Metadata directory not provided
I20260812 02:32:30.727403 24443 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:32:30.731691 24443 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260812 02:32:30.734125 24461 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:32:30.735782 24443 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 02:32:30.736004 24443 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501898713699-14748-0/minicluster-data/master-0/data,/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501898713699-14748-0/minicluster-data/master-0/wal
uuid: "9d1cf575f9e24f5c8b91051fc00f95a0"
format_stamp: "Formatted at 2026-08-12 02:32:29 on dist-test-slave-2w3w"
server_key: "4a9a6f9c957347e3029d14f9ae35072c"
server_key_iv: "9a057ec6fafbb916780fa602a38795f1"
server_key_version: "encryptionkey@0"
I20260812 02:32:30.736651 24443 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501898713699-14748-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501898713699-14748-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501898713699-14748-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:32:30.754707 24443 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:32:30.758653 24443 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:32:30.758931 24443 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:32:30.767539 24443 rpc_server.cc:307] RPC server started. Bound to: 127.14.103.62:44127
I20260812 02:32:30.767566 24520 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.103.62:44127 every 8 connection(s)
I20260812 02:32:30.768692 24443 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501898713699-14748-0/minicluster-data/master-0/data/info.pb
I20260812 02:32:30.773298 24521 sys_catalog.cc:263] Verifying existing consensus state
I20260812 02:32:30.775223 24521 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d1cf575f9e24f5c8b91051fc00f95a0: Bootstrap starting.
I20260812 02:32:30.775993 14748 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbep4M4/build/debug/bin/kudu as pid 24443
I20260812 02:32:30.776523 14748 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbep4M4/build/debug/bin/kudu
/tmp/dist-test-taskbep4M4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501898713699-14748-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501898713699-14748-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501898713699-14748-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501898713699-14748-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.103.1:43433
--local_ip_for_outbound_sockets=127.14.103.1
--tserver_master_addrs=127.14.103.62:44127
--webserver_port=40803
--webserver_interface=127.14.103.1
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.1.keytab
--principal=kudu/127.14.103.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.103.20:42525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.14.103.1
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501898713699-14748-0/krb5kdc/oryx_127.14.103.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501898713699-14748-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501898713699-14748-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501898713699-14748-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 02:32:30.794334 24521 log.cc:824] T 00000000000000000000000000000000 P 9d1cf575f9e24f5c8b91051fc00f95a0: Log is configured to *not* fsync() on all Append() calls
I20260812 02:32:30.802459 24521 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d1cf575f9e24f5c8b91051fc00f95a0: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 02:32:30.803052 24521 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d1cf575f9e24f5c8b91051fc00f95a0: Bootstrap complete.
I20260812 02:32:30.808610 24521 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9d1cf575f9e24f5c8b91051fc00f95a0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d1cf575f9e24f5c8b91051fc00f95a0" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 44127 } }
I20260812 02:32:30.809120 24521 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9d1cf575f9e24f5c8b91051fc00f95a0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d1cf575f9e24f5c8b91051fc00f95a0, State: Initialized, Role: FOLLOWER
I20260812 02:32:30.809751 24521 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9d1cf575f9e24f5c8b91051fc00f95a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d1cf575f9e24f5c8b91051fc00f95a0" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 44127 } }
I20260812 02:32:30.809893 24521 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9d1cf575f9e24f5c8b91051fc00f95a0 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 02:32:30.809993 24521 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9d1cf575f9e24f5c8b91051fc00f95a0 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 02:32:30.810120 24521 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9d1cf575f9e24f5c8b91051fc00f95a0 [term 1 FOLLOWER]: Advancing to term 2
I20260812 02:32:30.812624 24521 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9d1cf575f9e24f5c8b91051fc00f95a0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d1cf575f9e24f5c8b91051fc00f95a0" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 44127 } }
I20260812 02:32:30.813014 24521 leader_election.cc:302] T 00000000000000000000000000000000 P 9d1cf575f9e24f5c8b91051fc00f95a0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9d1cf575f9e24f5c8b91051fc00f95a0; no voters:
I20260812 02:32:30.813418 24521 leader_election.cc:288] T 00000000000000000000000000000000 P 9d1cf575f9e24f5c8b91051fc00f95a0 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260812 02:32:30.813546 24525 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9d1cf575f9e24f5c8b91051fc00f95a0 [term 2 FOLLOWER]: Leader election won for term 2
I20260812 02:32:30.813789 24525 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9d1cf575f9e24f5c8b91051fc00f95a0 [term 2 LEADER]: Becoming Leader. State: Replica: 9d1cf575f9e24f5c8b91051fc00f95a0, State: Running, Role: LEADER
I20260812 02:32:30.814119 24525 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9d1cf575f9e24f5c8b91051fc00f95a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d1cf575f9e24f5c8b91051fc00f95a0" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 44127 } }
I20260812 02:32:30.814939 24521 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9d1cf575f9e24f5c8b91051fc00f95a0 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:32:30.815912 24526 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d1cf575f9e24f5c8b91051fc00f95a0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "9d1cf575f9e24f5c8b91051fc00f95a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d1cf575f9e24f5c8b91051fc00f95a0" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 44127 } } }
I20260812 02:32:30.816063 24526 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d1cf575f9e24f5c8b91051fc00f95a0 [sys.catalog]: This master's current role is: LEADER
I20260812 02:32:30.816103 24527 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d1cf575f9e24f5c8b91051fc00f95a0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9d1cf575f9e24f5c8b91051fc00f95a0. Latest consensus state: current_term: 2 leader_uuid: "9d1cf575f9e24f5c8b91051fc00f95a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d1cf575f9e24f5c8b91051fc00f95a0" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 44127 } } }
I20260812 02:32:30.816222 24527 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d1cf575f9e24f5c8b91051fc00f95a0 [sys.catalog]: This master's current role is: LEADER
I20260812 02:32:30.816689 24535 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 02:32:30.821336 24535 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 02:32:30.822273 24535 catalog_manager.cc:1259] Loaded cluster ID: 5c79fdec80b2439eb3020cb7d9d58bb2
I20260812 02:32:30.822358 24535 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 02:32:30.824316 24535 catalog_manager.cc:1504] Loading token signing keys...
I20260812 02:32:30.824994 24535 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 9d1cf575f9e24f5c8b91051fc00f95a0: Loaded TSK: 0
I20260812 02:32:30.826030 24535 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260812 02:32:30.916065 24523 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:32:30.916378 24523 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:32:30.916450 24523 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:32:30.920913 24523 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:32:30.920996 24523 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:32:30.921062 24523 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.103.1
I20260812 02:32:30.926843 24523 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.103.20:42525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501898713699-14748-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501898713699-14748-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501898713699-14748-0/krb5kdc/oryx_127.14.103.1.keytab
--principal=oryx/127.14.103.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.103.1:43433
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501898713699-14748-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.103.1
--webserver_port=40803
--tserver_master_addrs=127.14.103.62:44127
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24523
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.103.1
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501898713699-14748-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:32:30.928047 24523 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:32:30.929152 24523 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:32:30.941004 24552 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:32:30.941022 24551 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:32:30.941236 24554 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:32:30.941617 24523 server_base.cc:1034] running on GCE node
I20260812 02:32:30.942137 24523 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:32:30.943059 24523 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:32:30.944293 24523 hybrid_clock.cc:648] HybridClock initialized: now 1786501950944256 us; error 48 us; skew 500 ppm
Aug 12 02:32:30 dist-test-slave-2w3w krb5kdc[23865](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501950, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.103.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:32:30.948190 24523 init.cc:377] Logged in from keytab as oryx/127.14.103.1@KRBTEST.COM (short username oryx)
I20260812 02:32:30.949512 24523 webserver.cc:460] Webserver started at http://127.14.103.1:40803/ using document root <none> and password file <none>
I20260812 02:32:30.950343 24523 fs_manager.cc:359] Metadata directory not provided
I20260812 02:32:30.950487 24523 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:32:30.956490 24523 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.003s
I20260812 02:32:30.959812 24562 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:32:30.961758 24523 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260812 02:32:30.961947 24523 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501898713699-14748-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501898713699-14748-0/minicluster-data/ts-0/wal
uuid: "c0bd44166b62428fa1941d3589cb060c"
format_stamp: "Formatted at 2026-08-12 02:32:29 on dist-test-slave-2w3w"
server_key: "43ac2b9b2ccdfef3f20298e6be7c2dec"
server_key_iv: "c51f4d450ac57f0f433bab6953d810ee"
server_key_version: "encryptionkey@0"
I20260812 02:32:30.962813 24523 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501898713699-14748-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501898713699-14748-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501898713699-14748-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:32:30.996861 24523 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:32:31.001313 24523 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:32:31.001552 24523 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:32:31.002331 24523 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:32:31.003726 24523 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:32:31.003826 24523 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:32:31.004446 24523 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:32:31.004516 24523 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:32:31.019862 24523 rpc_server.cc:307] RPC server started. Bound to: 127.14.103.1:43433
I20260812 02:32:31.019924 24696 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.103.1:43433 every 8 connection(s)
I20260812 02:32:31.021080 24523 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501898713699-14748-0/minicluster-data/ts-0/data/info.pb
I20260812 02:32:31.026870 14748 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbep4M4/build/debug/bin/kudu as pid 24523
I20260812 02:32:31.028014 14748 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbep4M4/build/debug/bin/kudu
/tmp/dist-test-taskbep4M4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501898713699-14748-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501898713699-14748-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501898713699-14748-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501898713699-14748-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.103.2:34345
--local_ip_for_outbound_sockets=127.14.103.2
--tserver_master_addrs=127.14.103.62:44127
--webserver_port=34421
--webserver_interface=127.14.103.2
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.2.keytab
--principal=kudu/127.14.103.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.103.20:42525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.14.103.2
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501898713699-14748-0/krb5kdc/oryx_127.14.103.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501898713699-14748-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501898713699-14748-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501898713699-14748-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 02:32:31 dist-test-slave-2w3w krb5kdc[23865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.103.1@KRBTEST.COM for oryx/127.14.103.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 02:32:31 dist-test-slave-2w3w krb5kdc[23865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.103.1@KRBTEST.COM for oryx/127.14.103.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 02:32:31.042654 24701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:31.024599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:44127 (local address 127.14.103.1:58873)
0812 02:32:31.025409 (+ 810us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:31.025460 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0812 02:32:31.027114 (+ 1654us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:32:31.029564 (+ 2450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:32:31.029578 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:32:31.030214 (+ 636us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:32:31.031087 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:31.031120 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:31.032921 (+ 1801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:31.032928 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:31.034700 (+ 1772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:31.034723 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:31.035959 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:31.035977 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:31.036283 (+ 306us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:31.037134 (+ 851us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:32:31.037164 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:32:31.041247 (+ 4083us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.103.62:44127: Server oryx/127.14.103.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":582,"thread_start_us":152,"threads_started":1}
W20260812 02:32:31.044063 24700 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:32:31.024106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.1:58873 (local address 127.14.103.62:44127)
0812 02:32:31.024743 (+ 637us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:31.024766 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:31.025633 (+ 867us) server_negotiation.cc:407] Connection header received
0812 02:32:31.028492 (+ 2859us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:31.028506 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:31.028888 (+ 382us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:31.029299 (+ 411us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:32:31.031374 (+ 2075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:31.032689 (+ 1315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:31.035028 (+ 2339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:31.035702 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:31.035885 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:31.042322 (+ 6437us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.103.1:58873: BlockingRecv error: recv got EOF from 127.14.103.1:58873 (error 108)
Metrics: {"server-negotiator.queue_time_us":391,"thread_start_us":184,"threads_started":1}
W20260812 02:32:31.044183 24697 heartbeater.cc:644] Failed to heartbeat to 127.14.103.62:44127 (0 consecutive failures): Not authorized: Failed to ping master at 127.14.103.62:44127: Client connection negotiation failed: client connection to 127.14.103.62:44127: Server oryx/127.14.103.62@KRBTEST.COM not found in Kerberos database
Aug 12 02:32:31 dist-test-slave-2w3w krb5kdc[23865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.103.1@KRBTEST.COM for oryx/127.14.103.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 02:32:31 dist-test-slave-2w3w krb5kdc[23865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.103.1@KRBTEST.COM for oryx/127.14.103.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 02:32:31.052618 24701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:31.044839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:44127 (local address 127.14.103.1:35693)
0812 02:32:31.045163 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:31.045182 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 02:32:31.045300 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:32:31.045679 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:32:31.045687 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:32:31.045947 (+ 260us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:32:31.046133 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:31.046146 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:31.047335 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:31.047342 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:31.048417 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:31.048435 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:31.049126 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:31.049133 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:31.049282 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:31.049945 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:32:31.049960 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:32:31.052446 (+ 2486us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.103.62:44127: Server oryx/127.14.103.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":209}
W20260812 02:32:31.052764 24700 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:32:31.045101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.1:35693 (local address 127.14.103.62:44127)
0812 02:32:31.045245 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:31.045249 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:31.045268 (+ 19us) server_negotiation.cc:407] Connection header received
0812 02:32:31.045416 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:31.045419 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:31.045490 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:31.045579 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:32:31.046293 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:31.047191 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:31.048588 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:31.048971 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:31.049066 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:31.052638 (+ 3572us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.103.1:35693: BlockingRecv error: recv got EOF from 127.14.103.1:35693 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
Aug 12 02:32:31 dist-test-slave-2w3w krb5kdc[23865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.103.1@KRBTEST.COM for oryx/127.14.103.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 02:32:31 dist-test-slave-2w3w krb5kdc[23865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.103.1@KRBTEST.COM for oryx/127.14.103.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 02:32:31.060113 24701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:31.053510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:44127 (local address 127.14.103.1:54913)
0812 02:32:31.053780 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:31.053797 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 02:32:31.053893 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:32:31.054328 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:32:31.054331 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:32:31.054511 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:32:31.054607 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:31.054613 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:31.055877 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:31.055880 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:31.056490 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:31.056500 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:31.057124 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:31.057127 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:31.057192 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:31.057790 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:32:31.057805 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:32:31.059998 (+ 2193us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.103.62:44127: Server oryx/127.14.103.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":191}
W20260812 02:32:31.060279 24700 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:32:31.053638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.1:54913 (local address 127.14.103.62:44127)
0812 02:32:31.053764 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:31.053769 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:31.053851 (+ 82us) server_negotiation.cc:407] Connection header received
0812 02:32:31.053997 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:31.053999 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:31.054102 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:31.054193 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:32:31.054736 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:31.055734 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:31.056621 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:31.056999 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:31.057108 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:31.060159 (+ 3051us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.103.1:54913: BlockingRecv error: recv got EOF from 127.14.103.1:54913 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20260812 02:32:31.060457 24697 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260812 02:32:31.165565 24702 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:32:31.165920 24702 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:32:31.165968 24702 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:32:31.171352 24702 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:32:31.171465 24702 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:32:31.171597 24702 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.103.2
I20260812 02:32:31.177946 24702 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.103.20:42525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501898713699-14748-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501898713699-14748-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501898713699-14748-0/krb5kdc/oryx_127.14.103.2.keytab
--principal=oryx/127.14.103.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.103.2:34345
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501898713699-14748-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.103.2
--webserver_port=34421
--tserver_master_addrs=127.14.103.62:44127
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24702
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.103.2
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501898713699-14748-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:32:31.179317 24702 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:32:31.181069 24702 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:32:31.192958 24707 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:32:31.192960 24712 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:32:31.192960 24710 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:32:31.193256 24702 server_base.cc:1034] running on GCE node
I20260812 02:32:31.193902 24702 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:32:31.194667 24702 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:32:31.195878 24702 hybrid_clock.cc:648] HybridClock initialized: now 1786501951195859 us; error 48 us; skew 500 ppm
Aug 12 02:32:31 dist-test-slave-2w3w krb5kdc[23865](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501951, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.103.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:32:31.199409 24702 init.cc:377] Logged in from keytab as oryx/127.14.103.2@KRBTEST.COM (short username oryx)
I20260812 02:32:31.200480 24702 webserver.cc:460] Webserver started at http://127.14.103.2:34421/ using document root <none> and password file <none>
I20260812 02:32:31.201198 24702 fs_manager.cc:359] Metadata directory not provided
I20260812 02:32:31.201366 24702 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:32:31.205996 24702 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260812 02:32:31.208858 24719 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:32:31.210417 24702 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260812 02:32:31.210597 24702 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501898713699-14748-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501898713699-14748-0/minicluster-data/ts-1/wal
uuid: "3a778b3f58274d0ab008fdf38dc5ad9b"
format_stamp: "Formatted at 2026-08-12 02:32:30 on dist-test-slave-2w3w"
server_key: "f73de3436ea4c013d2fa257d369f4c76"
server_key_iv: "ecfca5d0e8a265d86e21118ed27f156c"
server_key_version: "encryptionkey@0"
I20260812 02:32:31.211390 24702 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501898713699-14748-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501898713699-14748-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501898713699-14748-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:32:31.232647 24702 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:32:31.236569 24702 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:32:31.236819 24702 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:32:31.237521 24702 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:32:31.238512 24702 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:32:31.238580 24702 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:32:31.238656 24702 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:32:31.238699 24702 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:32:31.249796 24702 rpc_server.cc:307] RPC server started. Bound to: 127.14.103.2:34345
I20260812 02:32:31.249812 24852 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.103.2:34345 every 8 connection(s)
I20260812 02:32:31.250882 24702 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501898713699-14748-0/minicluster-data/ts-1/data/info.pb
I20260812 02:32:31.259153 14748 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbep4M4/build/debug/bin/kudu as pid 24702
I20260812 02:32:31.259790 14748 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbep4M4/build/debug/bin/kudu
/tmp/dist-test-taskbep4M4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501898713699-14748-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501898713699-14748-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501898713699-14748-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501898713699-14748-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.103.3:41919
--local_ip_for_outbound_sockets=127.14.103.3
--tserver_master_addrs=127.14.103.62:44127
--webserver_port=35455
--webserver_interface=127.14.103.3
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.3.keytab
--principal=kudu/127.14.103.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.103.20:42525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.14.103.3
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501898713699-14748-0/krb5kdc/oryx_127.14.103.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501898713699-14748-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501898713699-14748-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501898713699-14748-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 02:32:31 dist-test-slave-2w3w krb5kdc[23865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.103.2@KRBTEST.COM for oryx/127.14.103.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 02:32:31 dist-test-slave-2w3w krb5kdc[23865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.103.2@KRBTEST.COM for oryx/127.14.103.62@KRBTEST.COM, Server not found in Kerberos database
W20260812 02:32:31.263265 24700 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:32:31.252877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.2:60743 (local address 127.14.103.62:44127)
0812 02:32:31.253052 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:31.253057 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:31.254099 (+ 1042us) server_negotiation.cc:407] Connection header received
0812 02:32:31.255133 (+ 1034us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:31.255136 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:31.255192 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:31.255288 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:32:31.256552 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:31.257097 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:31.258418 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:31.258646 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:31.258751 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:31.263129 (+ 4378us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.103.2:60743: BlockingRecv error: recv got EOF from 127.14.103.2:60743 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
I20260812 02:32:31.263963 24855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:31.253252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:44127 (local address 127.14.103.2:60743)
0812 02:32:31.253942 (+ 690us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:31.253985 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0812 02:32:31.254889 (+ 904us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:32:31.255481 (+ 592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:32:31.255487 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:32:31.255874 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:32:31.256385 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:31.256400 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:31.257303 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:31.257306 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:31.258274 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:31.258286 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:31.258924 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:31.258930 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:31.259196 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:31.259892 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:32:31.259919 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:32:31.262957 (+ 3038us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.103.62:44127: Server oryx/127.14.103.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":120,"threads_started":1}
W20260812 02:32:31.264456 24853 heartbeater.cc:644] Failed to heartbeat to 127.14.103.62:44127 (0 consecutive failures): Not authorized: Failed to ping master at 127.14.103.62:44127: Client connection negotiation failed: client connection to 127.14.103.62:44127: Server oryx/127.14.103.62@KRBTEST.COM not found in Kerberos database
Aug 12 02:32:31 dist-test-slave-2w3w krb5kdc[23865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.103.2@KRBTEST.COM for oryx/127.14.103.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 02:32:31 dist-test-slave-2w3w krb5kdc[23865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.103.2@KRBTEST.COM for oryx/127.14.103.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 02:32:31.271721 24855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:31.264978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:44127 (local address 127.14.103.2:44117)
0812 02:32:31.265219 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:31.265232 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 02:32:31.265327 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:32:31.265652 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:32:31.265654 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:32:31.265817 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:32:31.265922 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:31.265927 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:31.267138 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:31.267141 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:31.267725 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:31.267735 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:31.268423 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:31.268425 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:31.268498 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:31.269093 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:32:31.269106 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:32:31.271605 (+ 2499us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.103.62:44127: Server oryx/127.14.103.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":53}
W20260812 02:32:31.272009 24700 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:32:31.265070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.2:44117 (local address 127.14.103.62:44127)
0812 02:32:31.265178 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:31.265181 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:31.265318 (+ 137us) server_negotiation.cc:407] Connection header received
0812 02:32:31.265431 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:31.265437 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:31.265505 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:31.265610 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:32:31.266051 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:31.267002 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:31.267860 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:31.268283 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:31.268429 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:31.271843 (+ 3414us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.103.2:44117: BlockingRecv error: recv got EOF from 127.14.103.2:44117 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
Aug 12 02:32:31 dist-test-slave-2w3w krb5kdc[23865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.103.2@KRBTEST.COM for oryx/127.14.103.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 02:32:31 dist-test-slave-2w3w krb5kdc[23865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.103.2@KRBTEST.COM for oryx/127.14.103.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 02:32:31.278604 24855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:31.272546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:44127 (local address 127.14.103.2:54085)
0812 02:32:31.272762 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:31.272775 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 02:32:31.272844 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:32:31.273210 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:32:31.273213 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:32:31.273365 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:32:31.273459 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:31.273465 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:31.274571 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:31.274574 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:31.275243 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:31.275254 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:31.275843 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:31.275846 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:31.275915 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:31.276464 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:32:31.276477 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:32:31.278488 (+ 2011us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.103.62:44127: Server oryx/127.14.103.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":131}
W20260812 02:32:31.278824 24700 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:32:31.272668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.2:54085 (local address 127.14.103.62:44127)
0812 02:32:31.272831 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:31.272837 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:31.272857 (+ 20us) server_negotiation.cc:407] Connection header received
0812 02:32:31.272938 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:31.272944 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:31.273007 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:31.273109 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:32:31.273584 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:31.274412 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:31.275359 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:31.275726 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:31.275832 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:31.278689 (+ 2857us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.103.2:54085: BlockingRecv error: recv got EOF from 127.14.103.2:54085 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260812 02:32:31.278982 24853 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260812 02:32:31.385828 24856 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:32:31.386152 24856 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:32:31.386199 24856 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:32:31.390708 24856 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:32:31.390792 24856 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:32:31.390858 24856 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.103.3
I20260812 02:32:31.396680 24856 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.103.20:42525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501898713699-14748-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501898713699-14748-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501898713699-14748-0/krb5kdc/oryx_127.14.103.3.keytab
--principal=oryx/127.14.103.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.103.3:41919
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501898713699-14748-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.103.3
--webserver_port=35455
--tserver_master_addrs=127.14.103.62:44127
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24856
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.103.3
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501898713699-14748-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:32:31.397840 24856 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:32:31.398886 24856 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:32:31.409518 24865 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:32:31.409543 24856 server_base.cc:1034] running on GCE node
W20260812 02:32:31.409543 24862 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:32:31.409660 24863 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:32:31.410269 24856 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:32:31.411065 24856 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:32:31.412276 24856 hybrid_clock.cc:648] HybridClock initialized: now 1786501951412263 us; error 29 us; skew 500 ppm
Aug 12 02:32:31 dist-test-slave-2w3w krb5kdc[23865](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501951, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.103.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:32:31.415723 24856 init.cc:377] Logged in from keytab as oryx/127.14.103.3@KRBTEST.COM (short username oryx)
I20260812 02:32:31.417042 24856 webserver.cc:460] Webserver started at http://127.14.103.3:35455/ using document root <none> and password file <none>
I20260812 02:32:31.417683 24856 fs_manager.cc:359] Metadata directory not provided
I20260812 02:32:31.417850 24856 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:32:31.422536 24856 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260812 02:32:31.425282 24873 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:32:31.426900 24856 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260812 02:32:31.427255 24856 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501898713699-14748-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501898713699-14748-0/minicluster-data/ts-2/wal
uuid: "478dcc6d5fc242399cdff84b438b2919"
format_stamp: "Formatted at 2026-08-12 02:32:30 on dist-test-slave-2w3w"
server_key: "2554373c7eff74890547e80d20176c52"
server_key_iv: "ad3c4940c277e9f8219a74c9148f1a3b"
server_key_version: "encryptionkey@0"
I20260812 02:32:31.427924 24856 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501898713699-14748-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501898713699-14748-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501898713699-14748-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:32:31.450201 24856 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:32:31.453975 24856 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:32:31.454216 24856 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:32:31.454890 24856 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:32:31.455888 24856 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:32:31.455965 24856 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:32:31.456043 24856 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:32:31.456086 24856 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:32:31.465938 24856 rpc_server.cc:307] RPC server started. Bound to: 127.14.103.3:41919
I20260812 02:32:31.466009 25006 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.103.3:41919 every 8 connection(s)
I20260812 02:32:31.467087 24856 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786501898713699-14748-0/minicluster-data/ts-2/data/info.pb
I20260812 02:32:31.469080 14748 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbep4M4/build/debug/bin/kudu as pid 24856
Aug 12 02:32:31 dist-test-slave-2w3w krb5kdc[23865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.103.3@KRBTEST.COM for oryx/127.14.103.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 02:32:31 dist-test-slave-2w3w krb5kdc[23865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.103.3@KRBTEST.COM for oryx/127.14.103.62@KRBTEST.COM, Server not found in Kerberos database
W20260812 02:32:31.478899 24700 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:32:31.469101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.3:49241 (local address 127.14.103.62:44127)
0812 02:32:31.469309 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:31.469314 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:31.470688 (+ 1374us) server_negotiation.cc:407] Connection header received
0812 02:32:31.471738 (+ 1050us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:31.471741 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:31.471792 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:31.471885 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:32:31.473063 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:31.473566 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:31.474770 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:31.475069 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:31.475177 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:31.478758 (+ 3581us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.103.3:49241: BlockingRecv error: recv got EOF from 127.14.103.3:49241 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
I20260812 02:32:31.479488 25012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:31.470173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34694 (local address 127.14.103.62:44127)
0812 02:32:31.470450 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:31.470454 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:31.470482 (+ 28us) server_negotiation.cc:407] Connection header received
0812 02:32:31.470535 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:31.470537 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:31.470578 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:31.470681 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:32:31.471622 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:31.472162 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:31.473043 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:31.473281 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:31.473393 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:31.475132 (+ 1739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:32:31.475139 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:32:31.475154 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:32:31.475183 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:32:31.477118 (+ 1935us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:31.477645 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:31.477652 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:31.477658 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:31.477844 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:31.478144 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:31.478148 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:31.478149 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:31.478536 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:32:31.478746 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0812 02:32:31.479088 (+ 342us) server_negotiation.cc:299] Negotiation successful
0812 02:32:31.479373 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":200,"thread_start_us":75,"threads_started":1}
I20260812 02:32:31.480005 25011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:31.469617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:44127 (local address 127.14.103.3:49241)
0812 02:32:31.470538 (+ 921us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:31.470595 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0812 02:32:31.471532 (+ 937us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:32:31.472080 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:32:31.472088 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:32:31.472441 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:32:31.472918 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:31.472931 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:31.473700 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:31.473704 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:31.474624 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:31.474634 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:31.475237 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:31.475242 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:31.475509 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:31.476086 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:32:31.476104 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:32:31.478639 (+ 2535us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.103.62:44127: Server oryx/127.14.103.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":690,"thread_start_us":176,"threads_started":1}
W20260812 02:32:31.480561 25007 heartbeater.cc:644] Failed to heartbeat to 127.14.103.62:44127 (0 consecutive failures): Not authorized: Failed to ping master at 127.14.103.62:44127: Client connection negotiation failed: client connection to 127.14.103.62:44127: Server oryx/127.14.103.62@KRBTEST.COM not found in Kerberos database
Aug 12 02:32:31 dist-test-slave-2w3w krb5kdc[23865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.103.3@KRBTEST.COM for oryx/127.14.103.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 02:32:31 dist-test-slave-2w3w krb5kdc[23865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.103.3@KRBTEST.COM for oryx/127.14.103.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 02:32:31.487322 25011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:31.481073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:44127 (local address 127.14.103.3:55963)
0812 02:32:31.481410 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:31.481424 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 02:32:31.481513 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:32:31.481989 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:32:31.481992 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:32:31.482171 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:32:31.482277 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:31.482282 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:31.483219 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:31.483222 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:31.483779 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:31.483787 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:31.484397 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:31.484400 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:31.484485 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:31.484955 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:32:31.484966 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:32:31.487157 (+ 2191us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.103.62:44127: Server oryx/127.14.103.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":244}
W20260812 02:32:31.487411 25012 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:32:31.481184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.3:55963 (local address 127.14.103.62:44127)
0812 02:32:31.481291 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:31.481294 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:31.481487 (+ 193us) server_negotiation.cc:407] Connection header received
0812 02:32:31.481608 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:31.481611 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:31.481658 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:31.481754 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:32:31.482455 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:31.483096 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:31.483905 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:31.484250 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:31.484339 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:31.487292 (+ 2953us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.103.3:55963: BlockingRecv error: recv got EOF from 127.14.103.3:55963 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
Aug 12 02:32:31 dist-test-slave-2w3w krb5kdc[23865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.103.3@KRBTEST.COM for oryx/127.14.103.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 02:32:31 dist-test-slave-2w3w krb5kdc[23865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.103.3@KRBTEST.COM for oryx/127.14.103.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 02:32:31.493525 25011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:31.488011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:44127 (local address 127.14.103.3:57585)
0812 02:32:31.488151 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:31.488168 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 02:32:31.488257 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:32:31.488476 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:32:31.488479 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:32:31.488616 (+ 137us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:32:31.488692 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:31.488697 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:31.489582 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:31.489584 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:31.490154 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:31.490163 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:31.490630 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:31.490633 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:31.490691 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:31.491313 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:32:31.491325 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:32:31.493374 (+ 2049us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.103.62:44127: Server oryx/127.14.103.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":59}
W20260812 02:32:31.493709 25012 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:32:31.488113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.3:57585 (local address 127.14.103.62:44127)
0812 02:32:31.488287 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:31.488291 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:31.488306 (+ 15us) server_negotiation.cc:407] Connection header received
0812 02:32:31.488341 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:31.488343 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:31.488378 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:31.488445 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:32:31.488877 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:31.489467 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:31.490285 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:31.490528 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:31.490617 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:31.493580 (+ 2963us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.103.3:57585: BlockingRecv error: recv got EOF from 127.14.103.3:57585 (error 108)
Metrics: {"server-negotiator.queue_time_us":98}
W20260812 02:32:31.493949 25007 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Aug 12 02:32:32 dist-test-slave-2w3w krb5kdc[23865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.103.1@KRBTEST.COM for oryx/127.14.103.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 02:32:32 dist-test-slave-2w3w krb5kdc[23865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.103.1@KRBTEST.COM for oryx/127.14.103.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 02:32:32.067950 25013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:32.061160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:44127 (local address 127.14.103.1:35965)
0812 02:32:32.061542 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:32.061558 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 02:32:32.061693 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:32:32.062044 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:32:32.062050 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:32:32.062354 (+ 304us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:32:32.062489 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:32.062497 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:32.063547 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:32.063551 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:32.064176 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:32.064186 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:32.064737 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:32.064740 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:32.064812 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:32.065445 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:32:32.065457 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:32:32.067774 (+ 2317us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.103.62:44127: Server oryx/127.14.103.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":90,"threads_started":1}
W20260812 02:32:32.068204 25014 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:32:32.061233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.1:35965 (local address 127.14.103.62:44127)
0812 02:32:32.061544 (+ 311us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:32.061549 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:32.061650 (+ 101us) server_negotiation.cc:407] Connection header received
0812 02:32:32.061797 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:32.061805 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:32.061877 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:32.061976 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:32:32.062644 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:32.063388 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:32.064329 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:32.064597 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:32.064697 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:32.068043 (+ 3346us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.103.1:35965: BlockingRecv error: recv got EOF from 127.14.103.1:35965 (error 108)
Metrics: {"server-negotiator.queue_time_us":232,"thread_start_us":130,"threads_started":1}
Aug 12 02:32:32 dist-test-slave-2w3w krb5kdc[23865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.103.2@KRBTEST.COM for oryx/127.14.103.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 02:32:32 dist-test-slave-2w3w krb5kdc[23865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.103.2@KRBTEST.COM for oryx/127.14.103.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 02:32:32.286204 25015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:32.279833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:44127 (local address 127.14.103.2:57865)
0812 02:32:32.280128 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:32.280150 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0812 02:32:32.280250 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:32:32.280533 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:32:32.280535 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:32:32.280727 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:32:32.280825 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:32.280878 (+ 53us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:32.281786 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:32.281789 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:32.282431 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:32.282441 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:32.282997 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:32.283000 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:32.283072 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:32.283745 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:32:32.283763 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:32:32.286014 (+ 2251us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.103.62:44127: Server oryx/127.14.103.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":110,"threads_started":1}
W20260812 02:32:32.286283 25014 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:32:32.280023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.2:57865 (local address 127.14.103.62:44127)
0812 02:32:32.280197 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:32.280201 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:32.280218 (+ 17us) server_negotiation.cc:407] Connection header received
0812 02:32:32.280350 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:32.280352 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:32.280399 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:32.280491 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:32:32.281028 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:32.281653 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:32.282575 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:32.282829 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:32.282937 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:32.286161 (+ 3224us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.103.2:57865: BlockingRecv error: recv got EOF from 127.14.103.2:57865 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Aug 12 02:32:32 dist-test-slave-2w3w krb5kdc[23865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.103.3@KRBTEST.COM for oryx/127.14.103.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 02:32:32 dist-test-slave-2w3w krb5kdc[23865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.103.3@KRBTEST.COM for oryx/127.14.103.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 02:32:32.501225 25016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:32.494771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:44127 (local address 127.14.103.3:50071)
0812 02:32:32.495219 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:32.495233 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 02:32:32.495397 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:32:32.495693 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:32:32.495696 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:32:32.495937 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:32:32.496036 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:32.496042 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:32.496859 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:32.496865 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:32.497452 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:32.497461 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:32.497962 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:32.497966 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:32.498057 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:32.498664 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:32:32.498677 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:32:32.500922 (+ 2245us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.103.62:44127: Server oryx/127.14.103.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":124,"threads_started":1}
W20260812 02:32:32.501353 25014 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:32:32.494933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.3:50071 (local address 127.14.103.62:44127)
0812 02:32:32.495156 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:32.495161 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:32.495325 (+ 164us) server_negotiation.cc:407] Connection header received
0812 02:32:32.495511 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:32.495515 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:32.495571 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:32.495667 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:32:32.496176 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:32.496734 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:32.497587 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:32.497827 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:32.497943 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:32.501111 (+ 3168us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.103.3:50071: BlockingRecv error: recv got EOF from 127.14.103.3:50071 (error 108)
Metrics: {"server-negotiator.queue_time_us":113}
Aug 12 02:32:33 dist-test-slave-2w3w krb5kdc[23865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.103.1@KRBTEST.COM for oryx/127.14.103.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 02:32:33 dist-test-slave-2w3w krb5kdc[23865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.103.1@KRBTEST.COM for oryx/127.14.103.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 02:32:33.080958 25017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:33.069063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:44127 (local address 127.14.103.1:41775)
0812 02:32:33.069351 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:33.069370 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 02:32:33.069505 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:32:33.069838 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:32:33.069841 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:32:33.070090 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:32:33.070199 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:33.070205 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:33.071243 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:33.071246 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:33.071822 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:33.071831 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:33.072339 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:33.072342 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:33.072430 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:33.077904 (+ 5474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:32:33.077922 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:32:33.080722 (+ 2800us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.103.62:44127: Server oryx/127.14.103.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":103,"threads_started":1}
W20260812 02:32:33.081394 25018 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:32:33.069180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.1:41775 (local address 127.14.103.62:44127)
0812 02:32:33.069466 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:33.069470 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:33.069488 (+ 18us) server_negotiation.cc:407] Connection header received
0812 02:32:33.069603 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:33.069606 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:33.069664 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:33.069766 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:32:33.070359 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:33.071083 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:33.071959 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:33.072211 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:33.072330 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:33.080980 (+ 8650us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.103.1:41775: BlockingRecv error: recv got EOF from 127.14.103.1:41775 (error 108)
Metrics: {"server-negotiator.queue_time_us":203,"thread_start_us":137,"threads_started":1}
Aug 12 02:32:33 dist-test-slave-2w3w krb5kdc[23865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.103.2@KRBTEST.COM for oryx/127.14.103.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 02:32:33 dist-test-slave-2w3w krb5kdc[23865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.103.2@KRBTEST.COM for oryx/127.14.103.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 02:32:33.293538 25019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:33.287380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:44127 (local address 127.14.103.2:35419)
0812 02:32:33.287772 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:33.287787 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 02:32:33.287941 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:32:33.288266 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:32:33.288269 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:32:33.288464 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:32:33.288563 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:33.288568 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:33.289320 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:33.289323 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:33.289906 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:33.289915 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:33.290427 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:33.290430 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:33.290513 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:33.291211 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:32:33.291224 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:32:33.293333 (+ 2109us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.103.62:44127: Server oryx/127.14.103.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":101,"threads_started":1}
W20260812 02:32:33.293642 25018 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:32:33.287367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.2:35419 (local address 127.14.103.62:44127)
0812 02:32:33.287622 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:33.287627 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:33.287921 (+ 294us) server_negotiation.cc:407] Connection header received
0812 02:32:33.288037 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:33.288039 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:33.288092 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:33.288186 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:32:33.288678 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:33.289199 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:33.290055 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:33.290282 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:33.290383 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:33.293502 (+ 3119us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.103.2:35419: BlockingRecv error: recv got EOF from 127.14.103.2:35419 (error 108)
Metrics: {"server-negotiator.queue_time_us":134}
Aug 12 02:32:33 dist-test-slave-2w3w krb5kdc[23865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.103.3@KRBTEST.COM for oryx/127.14.103.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 02:32:33 dist-test-slave-2w3w krb5kdc[23865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.103.3@KRBTEST.COM for oryx/127.14.103.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 02:32:33.511047 25021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:33.502914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:44127 (local address 127.14.103.3:51113)
0812 02:32:33.503609 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:33.503623 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 02:32:33.503747 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:32:33.504293 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:32:33.504296 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:32:33.504497 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:32:33.504610 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:33.504615 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:33.505518 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:33.505521 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:33.506296 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:33.506316 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:33.507008 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:33.507012 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:33.507281 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:33.508008 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:32:33.508029 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:32:33.510784 (+ 2755us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.103.62:44127: Server oryx/127.14.103.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":595,"thread_start_us":273,"threads_started":1}
W20260812 02:32:33.511129 25018 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:32:33.503240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.3:51113 (local address 127.14.103.62:44127)
0812 02:32:33.503428 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:33.503432 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:33.503747 (+ 315us) server_negotiation.cc:407] Connection header received
0812 02:32:33.503922 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:33.503925 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:33.503976 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:33.504048 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:32:33.504776 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:33.505356 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:33.506578 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:33.506826 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:33.506965 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:33.511001 (+ 4036us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.103.3:51113: BlockingRecv error: recv got EOF from 127.14.103.3:51113 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Aug 12 02:32:34 dist-test-slave-2w3w krb5kdc[23865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.103.1@KRBTEST.COM for oryx/127.14.103.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 02:32:34 dist-test-slave-2w3w krb5kdc[23865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.103.1@KRBTEST.COM for oryx/127.14.103.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 02:32:34.094940 25022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:34.083561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:44127 (local address 127.14.103.1:51749)
0812 02:32:34.085901 (+ 2340us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:34.085932 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0812 02:32:34.086264 (+ 332us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:32:34.086742 (+ 478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:32:34.086745 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:32:34.087577 (+ 832us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:32:34.087943 (+ 366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:34.087972 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:34.090205 (+ 2233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:34.090219 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:34.090838 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:34.090852 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:34.091670 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:34.091674 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:34.091788 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:34.092476 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:32:34.092489 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:32:34.094800 (+ 2311us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.103.62:44127: Server oryx/127.14.103.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":2179,"thread_start_us":273,"threads_started":1}
W20260812 02:32:34.095461 25023 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:32:34.083598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.1:51749 (local address 127.14.103.62:44127)
0812 02:32:34.085283 (+ 1685us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:34.085289 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:34.086453 (+ 1164us) server_negotiation.cc:407] Connection header received
0812 02:32:34.086519 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:34.086523 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:34.086586 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:34.086703 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:32:34.088427 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:34.089938 (+ 1511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:34.091062 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:34.091321 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:34.091653 (+ 332us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:34.095337 (+ 3684us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.103.1:51749: BlockingRecv error: recv got EOF from 127.14.103.1:51749 (error 108)
Metrics: {"server-negotiator.queue_time_us":667,"thread_start_us":261,"threads_started":1}
Aug 12 02:32:34 dist-test-slave-2w3w krb5kdc[23865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.103.2@KRBTEST.COM for oryx/127.14.103.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 02:32:34 dist-test-slave-2w3w krb5kdc[23865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.103.2@KRBTEST.COM for oryx/127.14.103.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 02:32:34.302493 25024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:34.294603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:44127 (local address 127.14.103.2:40659)
0812 02:32:34.295063 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:34.295081 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 02:32:34.295236 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:32:34.295541 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:32:34.295544 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:32:34.295789 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:32:34.295894 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:34.295901 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:34.296699 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:34.296704 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:34.297299 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:34.297309 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:34.298036 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:34.298040 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:34.298141 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:34.298881 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:32:34.298895 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:32:34.302261 (+ 3366us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.103.62:44127: Server oryx/127.14.103.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":110,"threads_started":1}
W20260812 02:32:34.302652 25023 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:32:34.294780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.2:40659 (local address 127.14.103.62:44127)
0812 02:32:34.294999 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:34.295003 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:34.295179 (+ 176us) server_negotiation.cc:407] Connection header received
0812 02:32:34.295349 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:34.295355 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:34.295413 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:34.295516 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:32:34.296021 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:34.296577 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:34.297616 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:34.297861 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:34.298030 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:34.302530 (+ 4500us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.103.2:40659: BlockingRecv error: recv got EOF from 127.14.103.2:40659 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Aug 12 02:32:34 dist-test-slave-2w3w krb5kdc[23865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.103.3@KRBTEST.COM for oryx/127.14.103.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 02:32:34 dist-test-slave-2w3w krb5kdc[23865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.103.3@KRBTEST.COM for oryx/127.14.103.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 02:32:34.520026 25025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:34.512371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:44127 (local address 127.14.103.3:41589)
0812 02:32:34.512847 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:34.512869 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0812 02:32:34.513031 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:32:34.513354 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:32:34.513357 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:32:34.513603 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:32:34.513706 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:34.513712 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:34.514543 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:34.514546 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:34.515369 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:34.515382 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:34.516360 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:34.516364 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:34.516461 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:34.517211 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:32:34.517228 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:32:34.519884 (+ 2656us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.103.62:44127: Server oryx/127.14.103.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":155,"threads_started":1}
W20260812 02:32:34.520334 25023 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:32:34.512576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.3:41589 (local address 127.14.103.62:44127)
0812 02:32:34.512797 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:34.512802 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:34.512960 (+ 158us) server_negotiation.cc:407] Connection header received
0812 02:32:34.513172 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:34.513175 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:34.513222 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:34.513326 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:32:34.513851 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:34.514406 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:34.515601 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:34.516120 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:34.516341 (+ 221us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:34.520104 (+ 3763us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.103.3:41589: BlockingRecv error: recv got EOF from 127.14.103.3:41589 (error 108)
Metrics: {"server-negotiator.queue_time_us":102}
Aug 12 02:32:35 dist-test-slave-2w3w krb5kdc[23865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.103.1@KRBTEST.COM for oryx/127.14.103.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 02:32:35 dist-test-slave-2w3w krb5kdc[23865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.103.1@KRBTEST.COM for oryx/127.14.103.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 02:32:35.102123 25026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:35.096067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:44127 (local address 127.14.103.1:52883)
0812 02:32:35.096352 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:35.096367 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 02:32:35.096460 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:32:35.096713 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:32:35.096715 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:32:35.096902 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:32:35.097045 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:35.097051 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:35.097879 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:35.097882 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:35.098462 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:35.098471 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:35.099095 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:35.099098 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:35.099164 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:35.099716 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:32:35.099728 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:32:35.101994 (+ 2266us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.103.62:44127: Server oryx/127.14.103.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":96,"threads_started":1}
W20260812 02:32:35.102442 25027 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:32:35.096104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.1:52883 (local address 127.14.103.62:44127)
0812 02:32:35.096420 (+ 316us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:35.096424 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:35.096438 (+ 14us) server_negotiation.cc:407] Connection header received
0812 02:32:35.096555 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:35.096557 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:35.096600 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:35.096678 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:32:35.097187 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:35.097761 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:35.098602 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:35.098885 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:35.099075 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:35.102275 (+ 3200us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.103.1:52883: BlockingRecv error: recv got EOF from 127.14.103.1:52883 (error 108)
Metrics: {"server-negotiator.queue_time_us":250,"thread_start_us":89,"threads_started":1}
Aug 12 02:32:35 dist-test-slave-2w3w krb5kdc[23865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.103.2@KRBTEST.COM for oryx/127.14.103.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 02:32:35 dist-test-slave-2w3w krb5kdc[23865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.103.2@KRBTEST.COM for oryx/127.14.103.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 02:32:35.310914 25028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:35.303676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:44127 (local address 127.14.103.2:36379)
0812 02:32:35.303964 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:35.304200 (+ 236us) client_negotiation.cc:174] Beginning negotiation
0812 02:32:35.304353 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:32:35.304873 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:32:35.304878 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:32:35.305096 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:32:35.305198 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:35.305204 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:35.306097 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:35.306196 (+ 99us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:35.307009 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:35.307026 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:35.307723 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:35.307728 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:35.307856 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:35.308647 (+ 791us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:32:35.308664 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:32:35.310767 (+ 2103us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.103.62:44127: Server oryx/127.14.103.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":111,"threads_started":1}
W20260812 02:32:35.311285 25027 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:32:35.303983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.2:36379 (local address 127.14.103.62:44127)
0812 02:32:35.304128 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:35.304132 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:35.304349 (+ 217us) server_negotiation.cc:407] Connection header received
0812 02:32:35.304441 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:35.304444 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:35.304498 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:35.304575 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:32:35.305364 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:35.305949 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:35.307213 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:35.307484 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:35.307589 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:35.311154 (+ 3565us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.103.2:36379: BlockingRecv error: recv got EOF from 127.14.103.2:36379 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Aug 12 02:32:35 dist-test-slave-2w3w krb5kdc[23865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.103.3@KRBTEST.COM for oryx/127.14.103.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 02:32:35 dist-test-slave-2w3w krb5kdc[23865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.103.3@KRBTEST.COM for oryx/127.14.103.62@KRBTEST.COM, Server not found in Kerberos database
W20260812 02:32:35.528052 25027 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:32:35.521257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.3:45827 (local address 127.14.103.62:44127)
0812 02:32:35.521452 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:35.521457 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:35.521728 (+ 271us) server_negotiation.cc:407] Connection header received
0812 02:32:35.521829 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:35.521832 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:35.521881 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:35.521962 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:32:35.522434 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:35.523248 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:35.524319 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:35.524652 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:35.524810 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:35.527923 (+ 3113us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.103.3:45827: BlockingRecv error: recv got EOF from 127.14.103.3:45827 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
I20260812 02:32:35.528066 25030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:35.521082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:44127 (local address 127.14.103.3:45827)
0812 02:32:35.521597 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:35.521612 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 02:32:35.521756 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:32:35.521997 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:32:35.521999 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:32:35.522200 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:32:35.522298 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:35.522304 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:35.523430 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:35.523438 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:35.524158 (+ 720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:35.524171 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:35.524827 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:35.524830 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:35.524911 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:35.525464 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:32:35.525476 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:32:35.527746 (+ 2270us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.103.62:44127: Server oryx/127.14.103.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":170,"threads_started":1}
Aug 12 02:32:36 dist-test-slave-2w3w krb5kdc[23865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.103.1@KRBTEST.COM for oryx/127.14.103.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 02:32:36 dist-test-slave-2w3w krb5kdc[23865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.103.1@KRBTEST.COM for oryx/127.14.103.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 02:32:36.112502 25031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:36.103250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:44127 (local address 127.14.103.1:55785)
0812 02:32:36.103814 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:36.103840 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0812 02:32:36.104502 (+ 662us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:32:36.104969 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:32:36.104973 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:32:36.105450 (+ 477us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:32:36.105610 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:36.105619 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:36.106876 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:36.106879 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:36.107862 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:36.107891 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:36.108920 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:36.108924 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:36.109050 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:36.109866 (+ 816us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:32:36.109880 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:32:36.112322 (+ 2442us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.103.62:44127: Server oryx/127.14.103.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":90,"threads_started":1}
W20260812 02:32:36.112810 25032 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:32:36.103379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.1:55785 (local address 127.14.103.62:44127)
0812 02:32:36.103645 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:36.103648 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:36.104346 (+ 698us) server_negotiation.cc:407] Connection header received
0812 02:32:36.104666 (+ 320us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:36.104669 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:36.104727 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:36.104858 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:32:36.106153 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:36.106719 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:36.108257 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:36.108628 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:36.108923 (+ 295us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:36.112661 (+ 3738us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.103.1:55785: BlockingRecv error: recv got EOF from 127.14.103.1:55785 (error 108)
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":96,"threads_started":1}
Aug 12 02:32:36 dist-test-slave-2w3w krb5kdc[23865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.103.2@KRBTEST.COM for oryx/127.14.103.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 02:32:36 dist-test-slave-2w3w krb5kdc[23865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.103.2@KRBTEST.COM for oryx/127.14.103.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 02:32:36.318872 25033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:36.312030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:44127 (local address 127.14.103.2:44455)
0812 02:32:36.312365 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:36.312380 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 02:32:36.312472 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:32:36.312824 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:32:36.312826 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:32:36.313034 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:32:36.313132 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:36.313137 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:36.314047 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:36.314051 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:36.314756 (+ 705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:36.314767 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:36.315420 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:36.315426 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:36.315591 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:36.316455 (+ 864us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:32:36.316469 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:32:36.318725 (+ 2256us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.103.62:44127: Server oryx/127.14.103.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":120,"threads_started":1}
W20260812 02:32:36.319139 25032 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:32:36.312225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.2:44455 (local address 127.14.103.62:44127)
0812 02:32:36.312418 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:36.312430 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:36.312447 (+ 17us) server_negotiation.cc:407] Connection header received
0812 02:32:36.312580 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:36.312582 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:36.312632 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:36.312721 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:32:36.313322 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:36.313913 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:36.314910 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:36.315247 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:36.315347 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:36.319005 (+ 3658us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.103.2:44455: BlockingRecv error: recv got EOF from 127.14.103.2:44455 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
Aug 12 02:32:36 dist-test-slave-2w3w krb5kdc[23865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.103.3@KRBTEST.COM for oryx/127.14.103.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 02:32:36 dist-test-slave-2w3w krb5kdc[23865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.103.3@KRBTEST.COM for oryx/127.14.103.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 02:32:36.535995 25034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:36.529301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:44127 (local address 127.14.103.3:36225)
0812 02:32:36.529617 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:36.529631 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 02:32:36.529736 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:32:36.530017 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:32:36.530019 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:32:36.530222 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:32:36.530344 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:36.530350 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:36.531338 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:36.531341 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:36.532033 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:36.532045 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:36.532638 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:36.532641 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:36.532721 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:36.533403 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:32:36.533417 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:32:36.535858 (+ 2441us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.103.62:44127: Server oryx/127.14.103.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":121,"threads_started":1}
W20260812 02:32:36.536231 25032 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:32:36.529511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.3:36225 (local address 127.14.103.62:44127)
0812 02:32:36.529648 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:36.529652 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:36.529687 (+ 35us) server_negotiation.cc:407] Connection header received
0812 02:32:36.529831 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:36.529834 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:36.529887 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:36.529973 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:32:36.530495 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:36.531162 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:36.532215 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:36.532488 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:36.532588 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:36.536082 (+ 3494us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.103.3:36225: BlockingRecv error: recv got EOF from 127.14.103.3:36225 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Aug 12 02:32:37 dist-test-slave-2w3w krb5kdc[23865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.103.1@KRBTEST.COM for oryx/127.14.103.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 02:32:37 dist-test-slave-2w3w krb5kdc[23865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.103.1@KRBTEST.COM for oryx/127.14.103.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 02:32:37.121932 25035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:37.113595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:44127 (local address 127.14.103.1:42885)
0812 02:32:37.113966 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:37.114020 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0812 02:32:37.114149 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:32:37.114568 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:32:37.114571 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:32:37.114804 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:32:37.115161 (+ 357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:37.115171 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:37.116210 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:37.116216 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:37.117015 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:37.117029 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:37.117894 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:37.117902 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:37.118013 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:37.118761 (+ 748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:32:37.118777 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:32:37.121775 (+ 2998us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.103.62:44127: Server oryx/127.14.103.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":104,"threads_started":1}
W20260812 02:32:37.122629 25036 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:32:37.113775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.1:42885 (local address 127.14.103.62:44127)
0812 02:32:37.114071 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:37.114075 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:37.114093 (+ 18us) server_negotiation.cc:407] Connection header received
0812 02:32:37.114305 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:37.114308 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:37.114360 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:37.114454 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:32:37.115329 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:37.116015 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:37.117302 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:37.117661 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:37.117781 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:37.122113 (+ 4332us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.103.1:42885: BlockingRecv error: recv got EOF from 127.14.103.1:42885 (error 108)
Metrics: {"server-negotiator.queue_time_us":225,"thread_start_us":89,"threads_started":1}
Aug 12 02:32:37 dist-test-slave-2w3w krb5kdc[23865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.103.2@KRBTEST.COM for oryx/127.14.103.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 02:32:37 dist-test-slave-2w3w krb5kdc[23865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.103.2@KRBTEST.COM for oryx/127.14.103.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 02:32:37.327158 25037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:37.319957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:44127 (local address 127.14.103.2:50567)
0812 02:32:37.320393 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:37.320413 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 02:32:37.320583 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:32:37.320892 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:32:37.320894 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:32:37.321140 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:32:37.321236 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:37.321242 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:37.322500 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:37.322504 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:37.323287 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:37.323298 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:37.323908 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:37.323915 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:37.324009 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:37.324635 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:32:37.324648 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:32:37.326962 (+ 2314us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.103.62:44127: Server oryx/127.14.103.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":131,"threads_started":1}
W20260812 02:32:37.327288 25036 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:32:37.320150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.2:50567 (local address 127.14.103.62:44127)
0812 02:32:37.320361 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:37.320366 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:37.320513 (+ 147us) server_negotiation.cc:407] Connection header received
0812 02:32:37.320696 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:37.320699 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:37.320760 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:37.320866 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:32:37.321537 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:37.322293 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:37.323399 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:37.323723 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:37.323859 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:37.327161 (+ 3302us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.103.2:50567: BlockingRecv error: recv got EOF from 127.14.103.2:50567 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
Aug 12 02:32:37 dist-test-slave-2w3w krb5kdc[23865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.103.3@KRBTEST.COM for oryx/127.14.103.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 02:32:37 dist-test-slave-2w3w krb5kdc[23865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.103.3@KRBTEST.COM for oryx/127.14.103.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 02:32:37.544106 25038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:37.537077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:44127 (local address 127.14.103.3:55167)
0812 02:32:37.537497 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:37.537530 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0812 02:32:37.537690 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:32:37.538037 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:32:37.538039 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:32:37.538268 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:32:37.538369 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:37.538375 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:37.539227 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:37.539231 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:37.539799 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:37.539807 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:37.540299 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:37.540302 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:37.540397 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:37.541111 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:32:37.541127 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:32:37.543906 (+ 2779us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.103.62:44127: Server oryx/127.14.103.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":102,"threads_started":1}
W20260812 02:32:37.544204 25036 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:32:37.537264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.3:55167 (local address 127.14.103.62:44127)
0812 02:32:37.537474 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:37.537479 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:37.537614 (+ 135us) server_negotiation.cc:407] Connection header received
0812 02:32:37.537803 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:37.537807 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:37.537906 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:37.538006 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:32:37.538489 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:37.539115 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:37.539925 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:37.540171 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:37.540289 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:37.544068 (+ 3779us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.103.3:55167: BlockingRecv error: recv got EOF from 127.14.103.3:55167 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
Aug 12 02:32:38 dist-test-slave-2w3w krb5kdc[23865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.103.1@KRBTEST.COM for oryx/127.14.103.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 02:32:38 dist-test-slave-2w3w krb5kdc[23865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.103.1@KRBTEST.COM for oryx/127.14.103.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 02:32:38.135377 25040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:38.123366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:44127 (local address 127.14.103.1:46243)
0812 02:32:38.123711 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:38.123731 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 02:32:38.123863 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:32:38.124202 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:32:38.124205 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:32:38.124406 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:32:38.124512 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:38.124518 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:38.125319 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:38.125322 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:38.125894 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:38.125903 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:38.126511 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:38.126514 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:38.126603 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:38.133032 (+ 6429us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:32:38.133050 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:32:38.135243 (+ 2193us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.103.62:44127: Server oryx/127.14.103.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":82,"threads_started":1}
W20260812 02:32:38.135684 25042 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:32:38.123557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.1:46243 (local address 127.14.103.62:44127)
0812 02:32:38.123833 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:38.123837 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:38.123853 (+ 16us) server_negotiation.cc:407] Connection header received
0812 02:32:38.123961 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:38.123963 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:38.124008 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:38.124102 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:32:38.124656 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:38.125210 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:38.126030 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:38.126355 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:38.126468 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:38.135533 (+ 9065us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.103.1:46243: BlockingRecv error: recv got EOF from 127.14.103.1:46243 (error 108)
Metrics: {"server-negotiator.queue_time_us":196,"thread_start_us":91,"threads_started":1}
Aug 12 02:32:38 dist-test-slave-2w3w krb5kdc[23865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.103.2@KRBTEST.COM for oryx/127.14.103.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 02:32:38 dist-test-slave-2w3w krb5kdc[23865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.103.2@KRBTEST.COM for oryx/127.14.103.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 02:32:38.335098 25043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:38.328357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:44127 (local address 127.14.103.2:54495)
0812 02:32:38.328806 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:38.328823 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 02:32:38.328965 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:32:38.329281 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:32:38.329284 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:32:38.329584 (+ 300us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:32:38.329687 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:38.329694 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:38.330575 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:38.330582 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:38.331289 (+ 707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:38.331299 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:38.331897 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:38.331901 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:38.332006 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:38.332646 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:32:38.332660 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:32:38.334864 (+ 2204us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.103.62:44127: Server oryx/127.14.103.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":81,"threads_started":1}
W20260812 02:32:38.335227 25042 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:32:38.328488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.2:54495 (local address 127.14.103.62:44127)
0812 02:32:38.328691 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:38.328697 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:38.328926 (+ 229us) server_negotiation.cc:407] Connection header received
0812 02:32:38.329092 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:38.329096 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:38.329149 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:38.329234 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:32:38.329832 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:38.330438 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:38.331431 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:38.331750 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:38.331858 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:38.335072 (+ 3214us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.103.2:54495: BlockingRecv error: recv got EOF from 127.14.103.2:54495 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
Aug 12 02:32:38 dist-test-slave-2w3w krb5kdc[23865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.103.3@KRBTEST.COM for oryx/127.14.103.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 02:32:38 dist-test-slave-2w3w krb5kdc[23865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.103.3@KRBTEST.COM for oryx/127.14.103.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 02:32:38.553746 25044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:38.545271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:44127 (local address 127.14.103.3:55357)
0812 02:32:38.545797 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:38.545816 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 02:32:38.545959 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:32:38.546520 (+ 561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:32:38.546524 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:32:38.546824 (+ 300us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:32:38.546958 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:38.546987 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:38.548325 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:38.548331 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:38.549187 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:38.549199 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:38.549946 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:38.549955 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:38.550097 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:38.550802 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:32:38.550824 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:32:38.553620 (+ 2796us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.103.62:44127: Server oryx/127.14.103.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":118,"threads_started":1}
W20260812 02:32:38.554090 25042 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:32:38.545495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.3:55357 (local address 127.14.103.62:44127)
0812 02:32:38.545656 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:38.545662 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:38.545926 (+ 264us) server_negotiation.cc:407] Connection header received
0812 02:32:38.546214 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:38.546221 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:38.546316 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:38.546455 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:32:38.547134 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:38.548086 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:38.549360 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:38.549750 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:38.550002 (+ 252us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:38.553924 (+ 3922us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.103.3:55357: BlockingRecv error: recv got EOF from 127.14.103.3:55357 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Aug 12 02:32:39 dist-test-slave-2w3w krb5kdc[23865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.103.1@KRBTEST.COM for oryx/127.14.103.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 02:32:39 dist-test-slave-2w3w krb5kdc[23865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.103.1@KRBTEST.COM for oryx/127.14.103.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 02:32:39.143647 25045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:39.136325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:44127 (local address 127.14.103.1:50489)
0812 02:32:39.136647 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:39.136662 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 02:32:39.136761 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:32:39.137017 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:32:39.137019 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:32:39.137208 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:32:39.137309 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:39.137314 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:39.138255 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:39.138261 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:39.139037 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:39.139048 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:39.139629 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:39.139633 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:39.139754 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:39.140619 (+ 865us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:32:39.140632 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:32:39.143403 (+ 2771us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.103.62:44127: Server oryx/127.14.103.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":136,"threads_started":1}
W20260812 02:32:39.143738 25046 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:32:39.136514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.1:50489 (local address 127.14.103.62:44127)
0812 02:32:39.136762 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:39.136766 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:39.136782 (+ 16us) server_negotiation.cc:407] Connection header received
0812 02:32:39.136853 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:39.136855 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:39.136897 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:39.136977 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:32:39.137447 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:39.138129 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:39.139162 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:39.139496 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:39.139611 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:39.143601 (+ 3990us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.103.1:50489: BlockingRecv error: recv got EOF from 127.14.103.1:50489 (error 108)
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":93,"threads_started":1}
Aug 12 02:32:39 dist-test-slave-2w3w krb5kdc[23865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.103.2@KRBTEST.COM for oryx/127.14.103.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 02:32:39 dist-test-slave-2w3w krb5kdc[23865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.103.2@KRBTEST.COM for oryx/127.14.103.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 02:32:39.343762 25047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:39.336210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:44127 (local address 127.14.103.2:47447)
0812 02:32:39.336547 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:39.336561 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 02:32:39.336694 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:32:39.337042 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:32:39.337046 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:32:39.337298 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:32:39.337429 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:39.337436 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:39.338673 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:39.338677 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:39.339300 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:39.339310 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:39.339823 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:39.339826 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:39.339892 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:39.340638 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:32:39.340651 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:32:39.343630 (+ 2979us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.103.62:44127: Server oryx/127.14.103.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":90,"threads_started":1}
W20260812 02:32:39.344048 25046 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:32:39.336348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.2:47447 (local address 127.14.103.62:44127)
0812 02:32:39.336547 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:39.336552 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:39.336643 (+ 91us) server_negotiation.cc:407] Connection header received
0812 02:32:39.336804 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:39.336807 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:39.336868 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:39.336953 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:32:39.337568 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:39.338417 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:39.339446 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:39.339676 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:39.339768 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:39.343910 (+ 4142us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.103.2:47447: BlockingRecv error: recv got EOF from 127.14.103.2:47447 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Aug 12 02:32:39 dist-test-slave-2w3w krb5kdc[23865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.103.3@KRBTEST.COM for oryx/127.14.103.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 02:32:39 dist-test-slave-2w3w krb5kdc[23865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.103.3@KRBTEST.COM for oryx/127.14.103.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 02:32:39.561556 25048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:39.554738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:44127 (local address 127.14.103.3:48197)
0812 02:32:39.555097 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:39.555112 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 02:32:39.555216 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:32:39.555530 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:32:39.555533 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:32:39.555728 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:32:39.555835 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:39.555841 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:39.556671 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:39.556674 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:39.557275 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:39.557285 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:39.558009 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:39.558012 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:39.558091 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:39.558668 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:32:39.558680 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:32:39.561335 (+ 2655us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.103.62:44127: Server oryx/127.14.103.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":97,"threads_started":1}
W20260812 02:32:39.561708 25046 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:32:39.554955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.3:48197 (local address 127.14.103.62:44127)
0812 02:32:39.555156 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:39.555159 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:39.555176 (+ 17us) server_negotiation.cc:407] Connection header received
0812 02:32:39.555339 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:39.555342 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:39.555391 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:39.555461 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:32:39.555976 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:39.556528 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:39.557409 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:39.557841 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:39.557946 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:39.561568 (+ 3622us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.103.3:48197: BlockingRecv error: recv got EOF from 127.14.103.3:48197 (error 108)
Metrics: {"server-negotiator.queue_time_us":96}
Aug 12 02:32:40 dist-test-slave-2w3w krb5kdc[23865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.103.1@KRBTEST.COM for oryx/127.14.103.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 02:32:40 dist-test-slave-2w3w krb5kdc[23865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.103.1@KRBTEST.COM for oryx/127.14.103.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 02:32:40.152011 25049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:40.144728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:44127 (local address 127.14.103.1:56307)
0812 02:32:40.145125 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:40.145147 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0812 02:32:40.145291 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:32:40.145595 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:32:40.145598 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:32:40.145933 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:32:40.146065 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:40.146072 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:40.147361 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:40.147366 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:40.148053 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:40.148064 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:40.148740 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:40.148743 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:40.148839 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:40.149613 (+ 774us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:32:40.149626 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:32:40.151801 (+ 2175us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.103.62:44127: Server oryx/127.14.103.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":140,"threads_started":1}
W20260812 02:32:40.152119 25050 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:32:40.144969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.1:56307 (local address 127.14.103.62:44127)
0812 02:32:40.145276 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:40.145280 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:40.145297 (+ 17us) server_negotiation.cc:407] Connection header received
0812 02:32:40.145424 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:40.145427 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:40.145474 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:40.145566 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:32:40.146202 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:40.147222 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:40.148188 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:40.148574 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:40.148720 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:40.151963 (+ 3243us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.103.1:56307: BlockingRecv error: recv got EOF from 127.14.103.1:56307 (error 108)
Metrics: {"server-negotiator.queue_time_us":207,"thread_start_us":88,"threads_started":1}
Aug 12 02:32:40 dist-test-slave-2w3w krb5kdc[23865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.103.2@KRBTEST.COM for oryx/127.14.103.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 02:32:40 dist-test-slave-2w3w krb5kdc[23865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.103.2@KRBTEST.COM for oryx/127.14.103.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 02:32:40.351176 25052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:40.344835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:44127 (local address 127.14.103.2:41177)
0812 02:32:40.345139 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:40.345153 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 02:32:40.345259 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:32:40.345570 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:32:40.345572 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:32:40.345760 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:32:40.345869 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:40.345874 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:40.346946 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:40.346953 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:40.347623 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:40.347634 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:40.348128 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:40.348132 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:40.348218 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:40.348805 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:32:40.348817 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:32:40.351013 (+ 2196us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.103.62:44127: Server oryx/127.14.103.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":73,"threads_started":1}
W20260812 02:32:40.351452 25050 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:32:40.345016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.2:41177 (local address 127.14.103.62:44127)
0812 02:32:40.345192 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:40.345196 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:40.345213 (+ 17us) server_negotiation.cc:407] Connection header received
0812 02:32:40.345355 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:40.345357 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:40.345423 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:40.345498 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:32:40.346017 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:40.346779 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:40.347759 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:40.348001 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:40.348120 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:40.351196 (+ 3076us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.103.2:41177: BlockingRecv error: recv got EOF from 127.14.103.2:41177 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Aug 12 02:32:40 dist-test-slave-2w3w krb5kdc[23865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.103.3@KRBTEST.COM for oryx/127.14.103.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 02:32:40 dist-test-slave-2w3w krb5kdc[23865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.103.3@KRBTEST.COM for oryx/127.14.103.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 02:32:40.569463 25053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:40.562667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:44127 (local address 127.14.103.3:57831)
0812 02:32:40.563162 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:40.563175 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 02:32:40.563325 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:32:40.563620 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:32:40.563623 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:32:40.563850 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:32:40.563998 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:40.564005 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:40.564789 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:40.564793 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:40.565390 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:40.565400 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:40.566045 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:40.566049 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:40.566147 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:40.566727 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:32:40.566739 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:32:40.569249 (+ 2510us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.103.62:44127: Server oryx/127.14.103.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":91,"threads_started":1}
W20260812 02:32:40.570327 25050 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:32:40.562902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.3:57831 (local address 127.14.103.62:44127)
0812 02:32:40.563072 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:40.563077 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:40.563259 (+ 182us) server_negotiation.cc:407] Connection header received
0812 02:32:40.563443 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:40.563448 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:40.563500 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:40.563597 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:32:40.564123 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:40.564669 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:40.565653 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:40.565912 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:40.566036 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:40.569494 (+ 3458us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.103.3:57831: BlockingRecv error: recv got EOF from 127.14.103.3:57831 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Aug 12 02:32:41 dist-test-slave-2w3w krb5kdc[23865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.103.1@KRBTEST.COM for oryx/127.14.103.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 02:32:41 dist-test-slave-2w3w krb5kdc[23865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.103.1@KRBTEST.COM for oryx/127.14.103.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 02:32:41.159881 25054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:41.153244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:44127 (local address 127.14.103.1:43707)
0812 02:32:41.153670 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:41.153691 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0812 02:32:41.153836 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:32:41.154130 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:32:41.154133 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:32:41.154437 (+ 304us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:32:41.154561 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:41.154567 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:41.155608 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:41.155613 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:41.156185 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:41.156194 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:41.156693 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:41.156697 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:41.156780 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:41.157382 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:32:41.157394 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:32:41.159642 (+ 2248us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.103.62:44127: Server oryx/127.14.103.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":89,"threads_started":1}
W20260812 02:32:41.160085 25055 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:32:41.153311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.1:43707 (local address 127.14.103.62:44127)
0812 02:32:41.153848 (+ 537us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:41.153853 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:41.153870 (+ 17us) server_negotiation.cc:407] Connection header received
0812 02:32:41.153927 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:41.153930 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:41.153980 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:41.154115 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:32:41.154692 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:41.155503 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:41.156313 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:41.156562 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:41.156675 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:41.159888 (+ 3213us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.103.1:43707: BlockingRecv error: recv got EOF from 127.14.103.1:43707 (error 108)
Metrics: {"server-negotiator.queue_time_us":455,"thread_start_us":299,"threads_started":1}
Aug 12 02:32:41 dist-test-slave-2w3w krb5kdc[23865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.103.2@KRBTEST.COM for oryx/127.14.103.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 02:32:41 dist-test-slave-2w3w krb5kdc[23865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.103.2@KRBTEST.COM for oryx/127.14.103.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 02:32:41.359146 25056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:41.352178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:44127 (local address 127.14.103.2:55627)
0812 02:32:41.352503 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:41.352518 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 02:32:41.352609 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:32:41.352977 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:32:41.352979 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:32:41.353182 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:32:41.353282 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:41.353291 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:41.354520 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:41.354526 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:41.355273 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:41.355284 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:41.355813 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:41.355825 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:41.355927 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:41.356639 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:32:41.356652 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:32:41.358940 (+ 2288us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.103.62:44127: Server oryx/127.14.103.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":128,"threads_started":1}
W20260812 02:32:41.359287 25055 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:32:41.352400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.2:55627 (local address 127.14.103.62:44127)
0812 02:32:41.352553 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:41.352557 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:41.352574 (+ 17us) server_negotiation.cc:407] Connection header received
0812 02:32:41.352749 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:41.352752 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:41.352805 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:41.352945 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:32:41.353537 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:41.354324 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:41.355404 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:41.355668 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:41.355797 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:41.359155 (+ 3358us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.103.2:55627: BlockingRecv error: recv got EOF from 127.14.103.2:55627 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Aug 12 02:32:41 dist-test-slave-2w3w krb5kdc[23865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.103.3@KRBTEST.COM for oryx/127.14.103.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 02:32:41 dist-test-slave-2w3w krb5kdc[23865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.103.3@KRBTEST.COM for oryx/127.14.103.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 02:32:41.578150 25057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:41.571093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:44127 (local address 127.14.103.3:45877)
0812 02:32:41.571474 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:41.571494 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 02:32:41.571618 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:32:41.571990 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:32:41.571994 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:32:41.572264 (+ 270us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:32:41.572365 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:41.572371 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:41.573216 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:41.573222 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:41.573859 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:41.573869 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:41.574399 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:41.574403 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:41.574499 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:41.575250 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:32:41.575262 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:32:41.577973 (+ 2711us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.103.62:44127: Server oryx/127.14.103.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":102,"threads_started":1}
W20260812 02:32:41.578277 25055 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:32:41.571290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.3:45877 (local address 127.14.103.62:44127)
0812 02:32:41.571477 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:41.571484 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:41.571560 (+ 76us) server_negotiation.cc:407] Connection header received
0812 02:32:41.571752 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:41.571756 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:41.571821 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:41.571916 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:32:41.572500 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:41.573092 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:41.574009 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:41.574264 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:41.574399 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:41.578160 (+ 3761us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.103.3:45877: BlockingRecv error: recv got EOF from 127.14.103.3:45877 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Aug 12 02:32:42 dist-test-slave-2w3w krb5kdc[23865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.103.1@KRBTEST.COM for oryx/127.14.103.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 02:32:42 dist-test-slave-2w3w krb5kdc[23865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.103.1@KRBTEST.COM for oryx/127.14.103.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 02:32:42.168567 25058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:42.161164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:44127 (local address 127.14.103.1:59799)
0812 02:32:42.161753 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:42.161771 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 02:32:42.161926 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:32:42.162298 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:32:42.162301 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:32:42.162533 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:32:42.162637 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:42.162644 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:42.163713 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:42.163716 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:42.164372 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:42.164384 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:42.165179 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:42.165183 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:42.165262 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:42.165865 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:32:42.165879 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:32:42.168351 (+ 2472us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.103.62:44127: Server oryx/127.14.103.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":134,"threads_started":1}
W20260812 02:32:42.168673 25059 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:32:42.161406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.1:59799 (local address 127.14.103.62:44127)
0812 02:32:42.161702 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:42.161707 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:42.161913 (+ 206us) server_negotiation.cc:407] Connection header received
0812 02:32:42.162047 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:42.162050 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:42.162111 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:42.162229 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:32:42.162792 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:42.163572 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:42.164521 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:42.164878 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:42.165054 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:42.168530 (+ 3476us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.103.1:59799: BlockingRecv error: recv got EOF from 127.14.103.1:59799 (error 108)
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":79,"threads_started":1}
Aug 12 02:32:42 dist-test-slave-2w3w krb5kdc[23865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.103.2@KRBTEST.COM for oryx/127.14.103.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 02:32:42 dist-test-slave-2w3w krb5kdc[23865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.103.2@KRBTEST.COM for oryx/127.14.103.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 02:32:42.366655 25060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:42.360061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:44127 (local address 127.14.103.2:48657)
0812 02:32:42.360396 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:42.360413 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 02:32:42.360544 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:32:42.360842 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:32:42.360848 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:32:42.361099 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:32:42.361240 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:42.361248 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:42.362190 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:42.362194 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:42.362777 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:42.362786 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:42.363543 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:42.363547 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:42.363634 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:42.364200 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:32:42.364212 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:32:42.366478 (+ 2266us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.103.62:44127: Server oryx/127.14.103.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":102,"threads_started":1}
W20260812 02:32:42.366775 25059 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:32:42.360123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.2:48657 (local address 127.14.103.62:44127)
0812 02:32:42.360402 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:42.360405 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:42.360506 (+ 101us) server_negotiation.cc:407] Connection header received
0812 02:32:42.360644 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:42.360647 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:42.360703 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:42.360788 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:32:42.361420 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:42.361991 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:42.362929 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:42.363352 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:42.363471 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:42.366651 (+ 3180us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.103.2:48657: BlockingRecv error: recv got EOF from 127.14.103.2:48657 (error 108)
Metrics: {"server-negotiator.queue_time_us":178}
Aug 12 02:32:42 dist-test-slave-2w3w krb5kdc[23865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.103.3@KRBTEST.COM for oryx/127.14.103.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 02:32:42 dist-test-slave-2w3w krb5kdc[23865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.103.3@KRBTEST.COM for oryx/127.14.103.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 02:32:42.585479 25061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:42.579228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:44127 (local address 127.14.103.3:48393)
0812 02:32:42.579597 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:42.579614 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 02:32:42.579744 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:32:42.580098 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:32:42.580101 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:32:42.580329 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:32:42.580424 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:42.580430 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:42.581224 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:42.581229 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:42.582051 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:42.582062 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:42.582558 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:42.582562 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:42.582649 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:42.583235 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:32:42.583248 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:32:42.585290 (+ 2042us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.103.62:44127: Server oryx/127.14.103.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":130,"threads_started":1}
W20260812 02:32:42.585585 25059 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:32:42.579423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.3:48393 (local address 127.14.103.62:44127)
0812 02:32:42.579598 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:42.579605 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:42.579687 (+ 82us) server_negotiation.cc:407] Connection header received
0812 02:32:42.579921 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:42.579924 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:42.579979 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:42.580075 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:32:42.580565 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:42.581106 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:42.582183 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:42.582423 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:42.582542 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:42.585464 (+ 2922us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.103.3:48393: BlockingRecv error: recv got EOF from 127.14.103.3:48393 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
Aug 12 02:32:43 dist-test-slave-2w3w krb5kdc[23865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.103.1@KRBTEST.COM for oryx/127.14.103.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 02:32:43 dist-test-slave-2w3w krb5kdc[23865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.103.1@KRBTEST.COM for oryx/127.14.103.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 02:32:43.182693 25063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:43.170304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:44127 (local address 127.14.103.1:45867)
0812 02:32:43.170606 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:43.170629 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0812 02:32:43.170740 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:32:43.171241 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:32:43.171243 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:32:43.171530 (+ 287us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:32:43.171679 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:43.171687 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:43.172522 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:43.172524 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:43.173089 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:43.173097 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:43.173651 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:43.173654 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:43.173725 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:43.179313 (+ 5588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:32:43.179332 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:32:43.181835 (+ 2503us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.103.62:44127: Server oryx/127.14.103.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":88,"threads_started":1}
W20260812 02:32:43.182912 25062 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:32:43.170153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.1:45867 (local address 127.14.103.62:44127)
0812 02:32:43.170517 (+ 364us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:43.170522 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:43.170691 (+ 169us) server_negotiation.cc:407] Connection header received
0812 02:32:43.170836 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:43.170839 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:43.171037 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:43.171199 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:32:43.171825 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:43.172401 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:43.173241 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:43.173497 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:43.173587 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:43.182736 (+ 9149us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.103.1:45867: BlockingRecv error: recv got EOF from 127.14.103.1:45867 (error 108)
Metrics: {"server-negotiator.queue_time_us":246,"thread_start_us":125,"threads_started":1}
Aug 12 02:32:43 dist-test-slave-2w3w krb5kdc[23865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.103.2@KRBTEST.COM for oryx/127.14.103.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 02:32:43 dist-test-slave-2w3w krb5kdc[23865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.103.2@KRBTEST.COM for oryx/127.14.103.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 02:32:43.374377 25064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:43.367687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:44127 (local address 127.14.103.2:45155)
0812 02:32:43.367984 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:43.367999 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 02:32:43.368102 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:32:43.368363 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:32:43.368365 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:32:43.368584 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:32:43.368686 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:43.368691 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:43.369834 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:43.369838 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:43.370550 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:43.370562 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:43.371326 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:43.371329 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:43.371398 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:43.372044 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:32:43.372057 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:32:43.374234 (+ 2177us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.103.62:44127: Server oryx/127.14.103.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":114,"threads_started":1}
W20260812 02:32:43.374784 25062 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:32:43.367851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.2:45155 (local address 127.14.103.62:44127)
0812 02:32:43.367997 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:43.368001 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:43.368054 (+ 53us) server_negotiation.cc:407] Connection header received
0812 02:32:43.368191 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:43.368194 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:43.368238 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:43.368331 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:32:43.368859 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:43.369668 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:43.370712 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:43.371148 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:43.371240 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:43.374644 (+ 3404us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.103.2:45155: BlockingRecv error: recv got EOF from 127.14.103.2:45155 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Aug 12 02:32:43 dist-test-slave-2w3w krb5kdc[23865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.103.3@KRBTEST.COM for oryx/127.14.103.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 02:32:43 dist-test-slave-2w3w krb5kdc[23865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.103.3@KRBTEST.COM for oryx/127.14.103.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 02:32:43.592947 25065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:43.586472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:44127 (local address 127.14.103.3:47995)
0812 02:32:43.586845 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:43.586863 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 02:32:43.587049 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:32:43.587353 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:32:43.587356 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:32:43.587598 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:32:43.587695 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:43.587700 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:43.588514 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:43.588520 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:43.589079 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:43.589088 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:43.589553 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:43.589556 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:43.589642 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:43.590444 (+ 802us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:32:43.590456 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:32:43.592766 (+ 2310us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.103.62:44127: Server oryx/127.14.103.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":148,"threads_started":1}
W20260812 02:32:43.593128 25062 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:32:43.586667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.3:47995 (local address 127.14.103.62:44127)
0812 02:32:43.586846 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:43.586852 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:43.586996 (+ 144us) server_negotiation.cc:407] Connection header received
0812 02:32:43.587173 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:43.587176 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:43.587234 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:43.587329 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:32:43.587823 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:43.588392 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:43.589206 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:43.589434 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:43.589548 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:43.592986 (+ 3438us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.103.3:47995: BlockingRecv error: recv got EOF from 127.14.103.3:47995 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Aug 12 02:32:44 dist-test-slave-2w3w krb5kdc[23865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.103.1@KRBTEST.COM for oryx/127.14.103.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 02:32:44 dist-test-slave-2w3w krb5kdc[23865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.103.1@KRBTEST.COM for oryx/127.14.103.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 02:32:44.191109 25066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:44.183908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:44127 (local address 127.14.103.1:49567)
0812 02:32:44.184341 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:44.184356 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 02:32:44.184447 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:32:44.184849 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:32:44.184853 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:32:44.185065 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:32:44.185179 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:44.185184 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:44.185949 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:44.185952 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:44.186539 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:44.186546 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:44.187082 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:44.187084 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:44.187148 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:44.187730 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:32:44.187743 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:32:44.190875 (+ 3132us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.103.62:44127: Server oryx/127.14.103.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":101,"threads_started":1}
W20260812 02:32:44.191411 25067 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:32:44.184036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.1:49567 (local address 127.14.103.62:44127)
0812 02:32:44.184562 (+ 526us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:44.184566 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:44.184578 (+ 12us) server_negotiation.cc:407] Connection header received
0812 02:32:44.184610 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:44.184613 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:44.184680 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:44.184776 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:32:44.185301 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:44.185853 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:44.186654 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:44.186894 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:44.187079 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:44.191267 (+ 4188us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.103.1:49567: BlockingRecv error: recv got EOF from 127.14.103.1:49567 (error 108)
Metrics: {"server-negotiator.queue_time_us":450,"thread_start_us":303,"threads_started":1}
Aug 12 02:32:44 dist-test-slave-2w3w krb5kdc[23865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.103.2@KRBTEST.COM for oryx/127.14.103.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 02:32:44 dist-test-slave-2w3w krb5kdc[23865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.103.2@KRBTEST.COM for oryx/127.14.103.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 02:32:44.382022 25068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:44.375365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:44127 (local address 127.14.103.2:58907)
0812 02:32:44.375646 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:44.375669 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0812 02:32:44.376034 (+ 365us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:32:44.376346 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:32:44.376349 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:32:44.376575 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:32:44.376671 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:44.376676 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:44.377450 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:44.377454 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:44.378041 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:44.378050 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:44.378552 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:44.378555 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:44.378641 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:44.379467 (+ 826us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:32:44.379482 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:32:44.381768 (+ 2286us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.103.62:44127: Server oryx/127.14.103.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":105,"threads_started":1}
W20260812 02:32:44.382046 25067 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:32:44.375603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.2:58907 (local address 127.14.103.62:44127)
0812 02:32:44.375957 (+ 354us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:44.375962 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:44.375983 (+ 21us) server_negotiation.cc:407] Connection header received
0812 02:32:44.376168 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:44.376172 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:44.376227 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:44.376320 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:32:44.376792 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:44.377330 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:44.378167 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:44.378393 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:44.378529 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:44.381939 (+ 3410us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.103.2:58907: BlockingRecv error: recv got EOF from 127.14.103.2:58907 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Aug 12 02:32:44 dist-test-slave-2w3w krb5kdc[23865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.103.3@KRBTEST.COM for oryx/127.14.103.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 02:32:44 dist-test-slave-2w3w krb5kdc[23865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.103.3@KRBTEST.COM for oryx/127.14.103.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 02:32:44.600118 25069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:44.593821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:44127 (local address 127.14.103.3:42591)
0812 02:32:44.594133 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:44.594146 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 02:32:44.594245 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:32:44.594456 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:32:44.594458 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:32:44.594705 (+ 247us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:32:44.594815 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:44.594822 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:44.595723 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:44.595727 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:44.596354 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:44.596364 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:44.596850 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:44.596852 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:44.596951 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:44.597555 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:32:44.597568 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:32:44.599940 (+ 2372us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.103.62:44127: Server oryx/127.14.103.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":140,"threads_started":1}
W20260812 02:32:44.600281 25067 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:32:44.593897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.3:42591 (local address 127.14.103.62:44127)
0812 02:32:44.594086 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:44.594090 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:44.594238 (+ 148us) server_negotiation.cc:407] Connection header received
0812 02:32:44.594320 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:44.594323 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:44.594369 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:44.594445 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:32:44.594932 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:44.595591 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:44.596502 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:44.596735 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:44.596826 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:44.600139 (+ 3313us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.103.3:42591: BlockingRecv error: recv got EOF from 127.14.103.3:42591 (error 108)
Metrics: {"server-negotiator.queue_time_us":94}
Aug 12 02:32:45 dist-test-slave-2w3w krb5kdc[23865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.103.1@KRBTEST.COM for oryx/127.14.103.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 02:32:45 dist-test-slave-2w3w krb5kdc[23865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.103.1@KRBTEST.COM for oryx/127.14.103.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 02:32:45.198374 25070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:45.192054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:44127 (local address 127.14.103.1:51869)
0812 02:32:45.192441 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:45.192458 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 02:32:45.192559 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:32:45.192809 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:32:45.192811 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:32:45.193020 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:32:45.193120 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:45.193126 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:45.194023 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:45.194026 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:45.194658 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:45.194668 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:45.195276 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:45.195280 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:45.195356 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:45.195990 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:32:45.196003 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:32:45.198224 (+ 2221us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.103.62:44127: Server oryx/127.14.103.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":99,"threads_started":1}
W20260812 02:32:45.198643 25071 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:32:45.192247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.1:51869 (local address 127.14.103.62:44127)
0812 02:32:45.192578 (+ 331us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:45.192582 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:45.192596 (+ 14us) server_negotiation.cc:407] Connection header received
0812 02:32:45.192639 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:45.192641 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:45.192687 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:45.192773 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:32:45.193273 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:45.193879 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:45.194807 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:45.195123 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:45.195218 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:45.198499 (+ 3281us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.103.1:51869: BlockingRecv error: recv got EOF from 127.14.103.1:51869 (error 108)
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":102,"threads_started":1}
Aug 12 02:32:45 dist-test-slave-2w3w krb5kdc[23865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.103.2@KRBTEST.COM for oryx/127.14.103.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 02:32:45 dist-test-slave-2w3w krb5kdc[23865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.103.2@KRBTEST.COM for oryx/127.14.103.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 02:32:45.389554 25072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:45.383440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:44127 (local address 127.14.103.2:55217)
0812 02:32:45.383829 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:45.383842 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 02:32:45.383969 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:32:45.384330 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:32:45.384340 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:32:45.384637 (+ 297us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:32:45.384766 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:45.384773 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:45.385617 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:45.385620 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:45.386266 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:45.386276 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:45.386789 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:45.386792 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:45.386860 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:45.387502 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:32:45.387514 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:32:45.389415 (+ 1901us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.103.62:44127: Server oryx/127.14.103.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":100,"threads_started":1}
W20260812 02:32:45.389757 25071 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:32:45.383501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.2:55217 (local address 127.14.103.62:44127)
0812 02:32:45.383668 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:45.383675 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:45.383926 (+ 251us) server_negotiation.cc:407] Connection header received
0812 02:32:45.384104 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:45.384107 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:45.384162 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:45.384262 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:32:45.384907 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:45.385479 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:45.386403 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:45.386638 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:45.386728 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:45.389615 (+ 2887us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.103.2:55217: BlockingRecv error: recv got EOF from 127.14.103.2:55217 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Aug 12 02:32:45 dist-test-slave-2w3w krb5kdc[23865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.103.3@KRBTEST.COM for oryx/127.14.103.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 02:32:45 dist-test-slave-2w3w krb5kdc[23865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.103.3@KRBTEST.COM for oryx/127.14.103.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 02:32:45.607627 25073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:45.601124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:44127 (local address 127.14.103.3:55415)
0812 02:32:45.601453 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:45.601468 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 02:32:45.601579 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:32:45.601909 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:32:45.601912 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:32:45.602114 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:32:45.602217 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:45.602222 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:45.603260 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:45.603263 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:45.604029 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:45.604041 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:45.604624 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:45.604627 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:45.604697 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:45.605225 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:32:45.605237 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:32:45.607467 (+ 2230us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.103.62:44127: Server oryx/127.14.103.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":122,"threads_started":1}
W20260812 02:32:45.607923 25071 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:32:45.601270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.3:55415 (local address 127.14.103.62:44127)
0812 02:32:45.601414 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:45.601421 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:45.601546 (+ 125us) server_negotiation.cc:407] Connection header received
0812 02:32:45.601679 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:45.601681 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:45.601732 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:45.601818 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:32:45.602353 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:45.603115 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:45.604182 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:45.604462 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:45.604548 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:45.607647 (+ 3099us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.103.3:55415: BlockingRecv error: recv got EOF from 127.14.103.3:55415 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Aug 12 02:32:46 dist-test-slave-2w3w krb5kdc[23865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.103.1@KRBTEST.COM for oryx/127.14.103.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 02:32:46 dist-test-slave-2w3w krb5kdc[23865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.103.1@KRBTEST.COM for oryx/127.14.103.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 02:32:46.206012 25074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:46.199581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:44127 (local address 127.14.103.1:36837)
0812 02:32:46.199873 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:46.199896 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0812 02:32:46.199993 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:32:46.200316 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:32:46.200319 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:32:46.200518 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:32:46.200621 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:46.200627 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:46.201520 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:46.201524 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:46.202191 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:46.202213 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:46.202789 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:46.202792 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:46.202865 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:46.203520 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:32:46.203533 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:32:46.205773 (+ 2240us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.103.62:44127: Server oryx/127.14.103.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":98,"threads_started":1}
W20260812 02:32:46.206178 25075 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:32:46.199707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.1:36837 (local address 127.14.103.62:44127)
0812 02:32:46.200030 (+ 323us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:46.200034 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:46.200050 (+ 16us) server_negotiation.cc:407] Connection header received
0812 02:32:46.200095 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:46.200097 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:46.200141 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:46.200236 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:32:46.200776 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:46.201368 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:46.202370 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:46.202640 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:46.202737 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:46.206040 (+ 3303us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.103.1:36837: BlockingRecv error: recv got EOF from 127.14.103.1:36837 (error 108)
Metrics: {"server-negotiator.queue_time_us":252,"thread_start_us":155,"threads_started":1}
Aug 12 02:32:46 dist-test-slave-2w3w krb5kdc[23865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.103.2@KRBTEST.COM for oryx/127.14.103.62@KRBTEST.COM, Server not found in Kerberos database
Aug 12 02:32:46 dist-test-slave-2w3w krb5kdc[23865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.103.2@KRBTEST.COM for oryx/127.14.103.62@KRBTEST.COM, Server not found in Kerberos database
I20260812 02:32:46.396277 25076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:46.390454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:44127 (local address 127.14.103.2:57939)
0812 02:32:46.390723 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:46.390736 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 02:32:46.390828 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:32:46.391096 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:32:46.391098 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:32:46.391288 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:32:46.391379 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:46.391384 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:46.392127 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:46.392129 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:46.392703 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:46.392711 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:46.393230 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:46.393233 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:46.393302 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:46.393842 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:32:46.393853 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:32:46.396139 (+ 2286us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.103.62:44127: Server oryx/127.14.103.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":87,"threads_started":1}
W20260812 02:32:46.396570 25075 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:32:46.390546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.2:57939 (local address 127.14.103.62:44127)
0812 02:32:46.390751 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:46.390754 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:46.390787 (+ 33us) server_negotiation.cc:407] Connection header received
0812 02:32:46.390917 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:46.390920 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:46.390995 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:46.391063 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:32:46.391501 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:46.392031 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:46.392852 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:46.393085 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:46.393186 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:46.396452 (+ 3266us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.103.2:57939: BlockingRecv error: recv got EOF from 127.14.103.2:57939 (error 108)
Metrics: {"server-negotiator.queue_time_us":104}
I20260812 02:32:46.473465 14748 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbep4M4/build/debug/bin/kudu with pid 24523
I20260812 02:32:46.481695 14748 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbep4M4/build/debug/bin/kudu with pid 24702
I20260812 02:32:46.489423 14748 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbep4M4/build/debug/bin/kudu with pid 24856
I20260812 02:32:46.497022 14748 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbep4M4/build/debug/bin/kudu with pid 24443
2026-08-12T02:32:46Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (18507 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-08-12T02:32:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T02:32:46Z Disabled control of system clock
I20260812 02:32:46.534404 14748 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbep4M4/build/debug/bin/kudu
/tmp/dist-test-taskbep4M4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786501898713699-14748-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786501898713699-14748-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786501898713699-14748-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786501898713699-14748-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.103.62:40695
--webserver_interface=127.14.103.62
--webserver_port=0
--builtin_ntp_servers=127.14.103.20:36693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.103.62:40695
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260812 02:32:46.657333 25080 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:32:46.657644 25080 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:32:46.657682 25080 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:32:46.662046 25080 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:32:46.662233 25080 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:32:46.662299 25080 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:32:46.662322 25080 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:32:46.662344 25080 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 02:32:46.667686 25080 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.103.20:36693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786501898713699-14748-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786501898713699-14748-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.103.62:40695
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.103.62:40695
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786501898713699-14748-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.103.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25080
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786501898713699-14748-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:32:46.668846 25080 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:32:46.670264 25080 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:32:46.679965 25086 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:32:46.680007 25089 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:32:46.680075 25080 server_base.cc:1034] running on GCE node
W20260812 02:32:46.680004 25087 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:32:46.681032 25080 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:32:46.682399 25080 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:32:46.683687 25080 hybrid_clock.cc:648] HybridClock initialized: now 1786501966683663 us; error 48 us; skew 500 ppm
I20260812 02:32:46.686128 25080 webserver.cc:460] Webserver started at http://127.14.103.62:32781/ using document root <none> and password file <none>
I20260812 02:32:46.686810 25080 fs_manager.cc:359] Metadata directory not provided
I20260812 02:32:46.686928 25080 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:32:46.687214 25080 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:32:46.689545 25080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786501898713699-14748-0/minicluster-data/master-0/data/instance:
uuid: "49979a3415834e479056250a2727e6d1"
format_stamp: "Formatted at 2026-08-12 02:32:46 on dist-test-slave-2w3w"
server_key: "dea684d179d9b0860a252e64bd247736"
server_key_iv: "b7789ad2e237d97b08b62e4f65a6533a"
server_key_version: "encryptionkey@0"
I20260812 02:32:46.690191 25080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786501898713699-14748-0/minicluster-data/master-0/wal/instance:
uuid: "49979a3415834e479056250a2727e6d1"
format_stamp: "Formatted at 2026-08-12 02:32:46 on dist-test-slave-2w3w"
server_key: "dea684d179d9b0860a252e64bd247736"
server_key_iv: "b7789ad2e237d97b08b62e4f65a6533a"
server_key_version: "encryptionkey@0"
I20260812 02:32:46.694716 25080 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260812 02:32:46.697598 25097 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:32:46.699090 25080 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 02:32:46.699237 25080 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786501898713699-14748-0/minicluster-data/master-0/data,/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786501898713699-14748-0/minicluster-data/master-0/wal
uuid: "49979a3415834e479056250a2727e6d1"
format_stamp: "Formatted at 2026-08-12 02:32:46 on dist-test-slave-2w3w"
server_key: "dea684d179d9b0860a252e64bd247736"
server_key_iv: "b7789ad2e237d97b08b62e4f65a6533a"
server_key_version: "encryptionkey@0"
I20260812 02:32:46.699380 25080 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786501898713699-14748-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786501898713699-14748-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786501898713699-14748-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:32:46.711143 25080 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:32:46.711973 25080 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:32:46.712184 25080 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:32:46.720003 25080 rpc_server.cc:307] RPC server started. Bound to: 127.14.103.62:40695
I20260812 02:32:46.720014 25158 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.103.62:40695 every 8 connection(s)
I20260812 02:32:46.721091 25080 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786501898713699-14748-0/minicluster-data/master-0/data/info.pb
I20260812 02:32:46.722106 14748 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbep4M4/build/debug/bin/kudu as pid 25080
I20260812 02:32:46.722230 14748 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786501898713699-14748-0/minicluster-data/master-0/wal/instance
I20260812 02:32:46.722571 14748 external_mini_cluster.cc:1428] Setting key f48caefb53f39aac200f044e970e5d1c
I20260812 02:32:46.725023 25159 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:32:46.732882 25159 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49979a3415834e479056250a2727e6d1: Bootstrap starting.
I20260812 02:32:46.734146 25163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:46.724273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44488 (local address 127.14.103.62:40695)
0812 02:32:46.725063 (+ 790us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:46.725092 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:46.725139 (+ 47us) server_negotiation.cc:407] Connection header received
0812 02:32:46.726042 (+ 903us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:46.726066 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:46.726575 (+ 509us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:46.727004 (+ 429us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:32:46.727456 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:46.728958 (+ 1502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:46.729958 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:46.730717 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:46.730897 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:46.731905 (+ 1008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:32:46.731912 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:32:46.731929 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:32:46.732464 (+ 535us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:32:46.732546 (+ 82us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:32:46.732573 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:32:46.732682 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0812 02:32:46.732865 (+ 183us) server_negotiation.cc:299] Negotiation successful
0812 02:32:46.733175 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":148,"threads_started":1}
I20260812 02:32:46.735679 25159 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 49979a3415834e479056250a2727e6d1: Neither blocks nor log segments found. Creating new log.
I20260812 02:32:46.736511 25159 log.cc:824] T 00000000000000000000000000000000 P 49979a3415834e479056250a2727e6d1: Log is configured to *not* fsync() on all Append() calls
I20260812 02:32:46.738688 25159 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49979a3415834e479056250a2727e6d1: No bootstrap required, opened a new log
I20260812 02:32:46.742406 25159 raft_consensus.cc:348] T 00000000000000000000000000000000 P 49979a3415834e479056250a2727e6d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49979a3415834e479056250a2727e6d1" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 40695 } }
I20260812 02:32:46.742713 25159 raft_consensus.cc:374] T 00000000000000000000000000000000 P 49979a3415834e479056250a2727e6d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:32:46.742800 25159 raft_consensus.cc:729] T 00000000000000000000000000000000 P 49979a3415834e479056250a2727e6d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49979a3415834e479056250a2727e6d1, State: Initialized, Role: FOLLOWER
I20260812 02:32:46.743320 25159 consensus_queue.cc:260] T 00000000000000000000000000000000 P 49979a3415834e479056250a2727e6d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49979a3415834e479056250a2727e6d1" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 40695 } }
I20260812 02:32:46.743490 25159 raft_consensus.cc:388] T 00000000000000000000000000000000 P 49979a3415834e479056250a2727e6d1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 02:32:46.743582 25159 raft_consensus.cc:482] T 00000000000000000000000000000000 P 49979a3415834e479056250a2727e6d1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 02:32:46.743690 25159 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 49979a3415834e479056250a2727e6d1 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:32:46.744787 25159 raft_consensus.cc:504] T 00000000000000000000000000000000 P 49979a3415834e479056250a2727e6d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49979a3415834e479056250a2727e6d1" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 40695 } }
I20260812 02:32:46.745167 25159 leader_election.cc:302] T 00000000000000000000000000000000 P 49979a3415834e479056250a2727e6d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 49979a3415834e479056250a2727e6d1; no voters:
I20260812 02:32:46.745532 25159 leader_election.cc:288] T 00000000000000000000000000000000 P 49979a3415834e479056250a2727e6d1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 02:32:46.745709 25165 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 49979a3415834e479056250a2727e6d1 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:32:46.746006 25165 raft_consensus.cc:686] T 00000000000000000000000000000000 P 49979a3415834e479056250a2727e6d1 [term 1 LEADER]: Becoming Leader. State: Replica: 49979a3415834e479056250a2727e6d1, State: Running, Role: LEADER
I20260812 02:32:46.746423 25165 consensus_queue.cc:237] T 00000000000000000000000000000000 P 49979a3415834e479056250a2727e6d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49979a3415834e479056250a2727e6d1" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 40695 } }
I20260812 02:32:46.747020 25159 sys_catalog.cc:564] T 00000000000000000000000000000000 P 49979a3415834e479056250a2727e6d1 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:32:46.748299 25168 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49979a3415834e479056250a2727e6d1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 49979a3415834e479056250a2727e6d1. Latest consensus state: current_term: 1 leader_uuid: "49979a3415834e479056250a2727e6d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49979a3415834e479056250a2727e6d1" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 40695 } } }
I20260812 02:32:46.748462 25168 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49979a3415834e479056250a2727e6d1 [sys.catalog]: This master's current role is: LEADER
I20260812 02:32:46.748822 25166 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49979a3415834e479056250a2727e6d1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "49979a3415834e479056250a2727e6d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49979a3415834e479056250a2727e6d1" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 40695 } } }
I20260812 02:32:46.748884 25176 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 02:32:46.748993 25166 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49979a3415834e479056250a2727e6d1 [sys.catalog]: This master's current role is: LEADER
I20260812 02:32:46.752388 25176 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 02:32:46.759222 25176 catalog_manager.cc:1347] Generated new cluster ID: f0f25305c97a4b7bb337c686b39fd5a8
I20260812 02:32:46.759339 25176 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 02:32:46.770380 25176 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 02:32:46.771760 25176 catalog_manager.cc:1504] Loading token signing keys...
I20260812 02:32:46.777671 25176 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 49979a3415834e479056250a2727e6d1: Generated new TSK 0
I20260812 02:32:46.778446 25176 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 02:32:46.782819 14748 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbep4M4/build/debug/bin/kudu
/tmp/dist-test-taskbep4M4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786501898713699-14748-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786501898713699-14748-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786501898713699-14748-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786501898713699-14748-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.103.1:0
--local_ip_for_outbound_sockets=127.14.103.1
--webserver_interface=127.14.103.1
--webserver_port=0
--tserver_master_addrs=127.14.103.62:40695
--builtin_ntp_servers=127.14.103.20:36693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260812 02:32:46.926086 25187 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:32:46.926436 25187 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:32:46.926479 25187 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:32:46.931180 25187 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:32:46.931259 25187 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:32:46.931320 25187 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.103.1
I20260812 02:32:46.936973 25187 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.103.20:36693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786501898713699-14748-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786501898713699-14748-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.103.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786501898713699-14748-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.103.1
--webserver_port=0
--tserver_master_addrs=127.14.103.62:40695
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25187
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.103.1
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786501898713699-14748-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:32:46.938191 25187 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:32:46.939667 25187 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:32:46.950271 25194 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:32:46.950273 25196 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:32:46.950271 25193 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:32:46.950840 25187 server_base.cc:1034] running on GCE node
I20260812 02:32:46.951442 25187 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:32:46.952172 25187 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:32:46.953354 25187 hybrid_clock.cc:648] HybridClock initialized: now 1786501966953371 us; error 120 us; skew 500 ppm
I20260812 02:32:46.955611 25187 webserver.cc:460] Webserver started at http://127.14.103.1:42327/ using document root <none> and password file <none>
I20260812 02:32:46.956277 25187 fs_manager.cc:359] Metadata directory not provided
I20260812 02:32:46.956385 25187 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:32:46.956704 25187 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:32:46.959352 25187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786501898713699-14748-0/minicluster-data/ts-0/data/instance:
uuid: "f54a1fc3a93049879cd3ada12761e0b2"
format_stamp: "Formatted at 2026-08-12 02:32:46 on dist-test-slave-2w3w"
server_key: "a8034c52fe2d9ea0a7b8a46bf60cf5af"
server_key_iv: "21c20e8efa77a34c5e14cb624b5228f1"
server_key_version: "encryptionkey@0"
I20260812 02:32:46.960045 25187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786501898713699-14748-0/minicluster-data/ts-0/wal/instance:
uuid: "f54a1fc3a93049879cd3ada12761e0b2"
format_stamp: "Formatted at 2026-08-12 02:32:46 on dist-test-slave-2w3w"
server_key: "a8034c52fe2d9ea0a7b8a46bf60cf5af"
server_key_iv: "21c20e8efa77a34c5e14cb624b5228f1"
server_key_version: "encryptionkey@0"
I20260812 02:32:46.966163 25187 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260812 02:32:46.969942 25203 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:32:46.971576 25187 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260812 02:32:46.971781 25187 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786501898713699-14748-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786501898713699-14748-0/minicluster-data/ts-0/wal
uuid: "f54a1fc3a93049879cd3ada12761e0b2"
format_stamp: "Formatted at 2026-08-12 02:32:46 on dist-test-slave-2w3w"
server_key: "a8034c52fe2d9ea0a7b8a46bf60cf5af"
server_key_iv: "21c20e8efa77a34c5e14cb624b5228f1"
server_key_version: "encryptionkey@0"
I20260812 02:32:46.971994 25187 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786501898713699-14748-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786501898713699-14748-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786501898713699-14748-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:32:46.985016 25187 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:32:46.985858 25187 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:32:46.986090 25187 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:32:46.986776 25187 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:32:46.987928 25187 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:32:46.988005 25187 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:32:46.988082 25187 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:32:46.988125 25187 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:32:47.001125 25187 rpc_server.cc:307] RPC server started. Bound to: 127.14.103.1:41417
I20260812 02:32:47.001168 25335 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.103.1:41417 every 8 connection(s)
I20260812 02:32:47.002521 25187 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786501898713699-14748-0/minicluster-data/ts-0/data/info.pb
I20260812 02:32:47.002837 14748 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbep4M4/build/debug/bin/kudu as pid 25187
I20260812 02:32:47.003110 14748 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786501898713699-14748-0/minicluster-data/ts-0/wal/instance
I20260812 02:32:47.003584 14748 external_mini_cluster.cc:1428] Setting key 82296678d407b48a8d928e41dc26df85
I20260812 02:32:47.007534 14748 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbep4M4/build/debug/bin/kudu
/tmp/dist-test-taskbep4M4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786501898713699-14748-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786501898713699-14748-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786501898713699-14748-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786501898713699-14748-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.103.2:0
--local_ip_for_outbound_sockets=127.14.103.2
--webserver_interface=127.14.103.2
--webserver_port=0
--tserver_master_addrs=127.14.103.62:40695
--builtin_ntp_servers=127.14.103.20:36693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260812 02:32:47.017449 25163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:47.005352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.1:34391 (local address 127.14.103.62:40695)
0812 02:32:47.005484 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:47.005487 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:47.007232 (+ 1745us) server_negotiation.cc:407] Connection header received
0812 02:32:47.010567 (+ 3335us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:47.010574 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:47.010800 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:47.011011 (+ 211us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:32:47.012209 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:47.013142 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:47.014666 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:47.015166 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:47.015312 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:47.016629 (+ 1317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:32:47.016636 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:32:47.016641 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:32:47.016663 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:32:47.016690 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:32:47.016704 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:32:47.016802 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0812 02:32:47.017099 (+ 297us) server_negotiation.cc:299] Negotiation successful
0812 02:32:47.017221 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260812 02:32:47.018358 25338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:47.005575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:40695 (local address 127.14.103.1:34391)
0812 02:32:47.006905 (+ 1330us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:47.007004 (+ 99us) client_negotiation.cc:174] Beginning negotiation
0812 02:32:47.010108 (+ 3104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:32:47.011312 (+ 1204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:32:47.011322 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:32:47.011446 (+ 124us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:32:47.012015 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:47.012029 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:47.013328 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:47.013333 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:47.014498 (+ 1165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:47.014511 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:47.015349 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:47.015352 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:47.015550 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:47.016285 (+ 735us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 02:32:47.016287 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:32:47.016335 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 02:32:47.016428 (+ 93us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:32:47.016829 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:32:47.016839 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:32:47.016844 (+ 5us) client_negotiation.cc:769] Sending connection context
0812 02:32:47.017067 (+ 223us) client_negotiation.cc:240] Negotiation successful
0812 02:32:47.017346 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":561,"thread_start_us":191,"threads_started":1}
I20260812 02:32:47.020035 25336 heartbeater.cc:347] Connected to a master server at 127.14.103.62:40695
I20260812 02:32:47.020306 25336 heartbeater.cc:464] Registering TS with master...
I20260812 02:32:47.020922 25336 heartbeater.cc:511] Master 127.14.103.62:40695 requested a full tablet report, sending...
I20260812 02:32:47.022933 25117 ts_manager.cc:194] Registered new tserver with Master: f54a1fc3a93049879cd3ada12761e0b2 (127.14.103.1:41417)
I20260812 02:32:47.024449 25117 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.103.1:34391
W20260812 02:32:47.140071 25339 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:32:47.140396 25339 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:32:47.140437 25339 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:32:47.144994 25339 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:32:47.145094 25339 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:32:47.145157 25339 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.103.2
I20260812 02:32:47.151302 25339 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.103.20:36693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786501898713699-14748-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786501898713699-14748-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.103.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786501898713699-14748-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.103.2
--webserver_port=0
--tserver_master_addrs=127.14.103.62:40695
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25339
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.103.2
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786501898713699-14748-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:32:47.152531 25339 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:32:47.154026 25339 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:32:47.165551 25346 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:32:47.165684 25348 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:32:47.165555 25345 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:32:47.166637 25339 server_base.cc:1034] running on GCE node
I20260812 02:32:47.167181 25339 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:32:47.168083 25339 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:32:47.169328 25339 hybrid_clock.cc:648] HybridClock initialized: now 1786501967169280 us; error 66 us; skew 500 ppm
I20260812 02:32:47.171787 25339 webserver.cc:460] Webserver started at http://127.14.103.2:33679/ using document root <none> and password file <none>
I20260812 02:32:47.172468 25339 fs_manager.cc:359] Metadata directory not provided
I20260812 02:32:47.172542 25339 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:32:47.172775 25339 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:32:47.175252 25339 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786501898713699-14748-0/minicluster-data/ts-1/data/instance:
uuid: "808572853851489984a62cae03cb0a98"
format_stamp: "Formatted at 2026-08-12 02:32:47 on dist-test-slave-2w3w"
server_key: "e6ff548e5ade98c86fc1684020edadb9"
server_key_iv: "d1c6da760b12045412734cdc6fb2f314"
server_key_version: "encryptionkey@0"
I20260812 02:32:47.175992 25339 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786501898713699-14748-0/minicluster-data/ts-1/wal/instance:
uuid: "808572853851489984a62cae03cb0a98"
format_stamp: "Formatted at 2026-08-12 02:32:47 on dist-test-slave-2w3w"
server_key: "e6ff548e5ade98c86fc1684020edadb9"
server_key_iv: "d1c6da760b12045412734cdc6fb2f314"
server_key_version: "encryptionkey@0"
I20260812 02:32:47.180606 25339 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260812 02:32:47.184412 25357 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:32:47.186069 25339 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 02:32:47.186259 25339 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786501898713699-14748-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786501898713699-14748-0/minicluster-data/ts-1/wal
uuid: "808572853851489984a62cae03cb0a98"
format_stamp: "Formatted at 2026-08-12 02:32:47 on dist-test-slave-2w3w"
server_key: "e6ff548e5ade98c86fc1684020edadb9"
server_key_iv: "d1c6da760b12045412734cdc6fb2f314"
server_key_version: "encryptionkey@0"
I20260812 02:32:47.186547 25339 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786501898713699-14748-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786501898713699-14748-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786501898713699-14748-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:32:47.222806 25339 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:32:47.223758 25339 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:32:47.224016 25339 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:32:47.224766 25339 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:32:47.226052 25339 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:32:47.226145 25339 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:32:47.226227 25339 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:32:47.226280 25339 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:32:47.238181 25339 rpc_server.cc:307] RPC server started. Bound to: 127.14.103.2:37633
I20260812 02:32:47.238222 25488 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.103.2:37633 every 8 connection(s)
I20260812 02:32:47.239284 25339 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786501898713699-14748-0/minicluster-data/ts-1/data/info.pb
I20260812 02:32:47.249666 14748 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbep4M4/build/debug/bin/kudu as pid 25339
I20260812 02:32:47.249773 14748 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786501898713699-14748-0/minicluster-data/ts-1/wal/instance
I20260812 02:32:47.250136 14748 external_mini_cluster.cc:1428] Setting key ccd57ea470f4b2e245eb426a0ac78793
I20260812 02:32:47.251894 25163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:47.242206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.2:34139 (local address 127.14.103.62:40695)
0812 02:32:47.242496 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:47.242501 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:47.243662 (+ 1161us) server_negotiation.cc:407] Connection header received
0812 02:32:47.245263 (+ 1601us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:47.245266 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:47.245329 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:47.245433 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:32:47.246483 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:47.247381 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:47.248813 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:47.249124 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:47.249253 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:47.251188 (+ 1935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:32:47.251192 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:32:47.251195 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:32:47.251295 (+ 100us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:32:47.251325 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:32:47.251343 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:32:47.251434 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0812 02:32:47.251669 (+ 235us) server_negotiation.cc:299] Negotiation successful
0812 02:32:47.251739 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118}
I20260812 02:32:47.252992 25492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:47.242568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:40695 (local address 127.14.103.2:34139)
0812 02:32:47.243464 (+ 896us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:47.243516 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0812 02:32:47.244934 (+ 1418us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:32:47.245637 (+ 703us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:32:47.245645 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:32:47.245744 (+ 99us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:32:47.246315 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:47.246328 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:47.247527 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:47.247530 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:47.248636 (+ 1106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:47.248648 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:47.249322 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:47.249325 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:47.249522 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:47.250531 (+ 1009us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 02:32:47.250534 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:32:47.250658 (+ 124us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 02:32:47.250921 (+ 263us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:32:47.251482 (+ 561us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:32:47.251494 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:32:47.251501 (+ 7us) client_negotiation.cc:769] Sending connection context
0812 02:32:47.251664 (+ 163us) client_negotiation.cc:240] Negotiation successful
0812 02:32:47.252049 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":149,"threads_started":1}
I20260812 02:32:47.253144 14748 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbep4M4/build/debug/bin/kudu
/tmp/dist-test-taskbep4M4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786501898713699-14748-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786501898713699-14748-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786501898713699-14748-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786501898713699-14748-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.103.3:0
--local_ip_for_outbound_sockets=127.14.103.3
--webserver_interface=127.14.103.3
--webserver_port=0
--tserver_master_addrs=127.14.103.62:40695
--builtin_ntp_servers=127.14.103.20:36693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260812 02:32:47.254913 25489 heartbeater.cc:347] Connected to a master server at 127.14.103.62:40695
I20260812 02:32:47.255457 25489 heartbeater.cc:464] Registering TS with master...
I20260812 02:32:47.256352 25489 heartbeater.cc:511] Master 127.14.103.62:40695 requested a full tablet report, sending...
I20260812 02:32:47.257766 25117 ts_manager.cc:194] Registered new tserver with Master: 808572853851489984a62cae03cb0a98 (127.14.103.2:37633)
I20260812 02:32:47.258430 25117 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.103.2:34139
W20260812 02:32:47.385308 25493 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:32:47.385640 25493 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:32:47.385682 25493 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:32:47.392086 25493 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:32:47.392211 25493 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:32:47.392323 25493 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.103.3
I20260812 02:32:47.398146 25493 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.103.20:36693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786501898713699-14748-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786501898713699-14748-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.103.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786501898713699-14748-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.103.3
--webserver_port=0
--tserver_master_addrs=127.14.103.62:40695
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25493
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.103.3
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786501898713699-14748-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:32:47.399418 25493 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:32:47.400892 25493 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:32:47.412667 25498 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:32:47.412667 25499 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:32:47.412678 25501 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:32:47.413003 25493 server_base.cc:1034] running on GCE node
I20260812 02:32:47.413640 25493 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:32:47.414455 25493 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:32:47.415671 25493 hybrid_clock.cc:648] HybridClock initialized: now 1786501967415656 us; error 43 us; skew 500 ppm
I20260812 02:32:47.417857 25493 webserver.cc:460] Webserver started at http://127.14.103.3:32941/ using document root <none> and password file <none>
I20260812 02:32:47.418516 25493 fs_manager.cc:359] Metadata directory not provided
I20260812 02:32:47.418598 25493 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:32:47.418833 25493 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:32:47.421202 25493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786501898713699-14748-0/minicluster-data/ts-2/data/instance:
uuid: "98baacd9865647899cad6ac4ba481a09"
format_stamp: "Formatted at 2026-08-12 02:32:47 on dist-test-slave-2w3w"
server_key: "f4da1e45844839bb455825484c9f7f87"
server_key_iv: "9530fa759041af5a6d68194a9fdcc573"
server_key_version: "encryptionkey@0"
I20260812 02:32:47.421861 25493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786501898713699-14748-0/minicluster-data/ts-2/wal/instance:
uuid: "98baacd9865647899cad6ac4ba481a09"
format_stamp: "Formatted at 2026-08-12 02:32:47 on dist-test-slave-2w3w"
server_key: "f4da1e45844839bb455825484c9f7f87"
server_key_iv: "9530fa759041af5a6d68194a9fdcc573"
server_key_version: "encryptionkey@0"
I20260812 02:32:47.426705 25493 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260812 02:32:47.430909 25510 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:32:47.432706 25493 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 02:32:47.432915 25493 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786501898713699-14748-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786501898713699-14748-0/minicluster-data/ts-2/wal
uuid: "98baacd9865647899cad6ac4ba481a09"
format_stamp: "Formatted at 2026-08-12 02:32:47 on dist-test-slave-2w3w"
server_key: "f4da1e45844839bb455825484c9f7f87"
server_key_iv: "9530fa759041af5a6d68194a9fdcc573"
server_key_version: "encryptionkey@0"
I20260812 02:32:47.433106 25493 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786501898713699-14748-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786501898713699-14748-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786501898713699-14748-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:32:47.453565 25493 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:32:47.454332 25493 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:32:47.454509 25493 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:32:47.455142 25493 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:32:47.456126 25493 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:32:47.456175 25493 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:32:47.456218 25493 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:32:47.456235 25493 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:32:47.466516 25493 rpc_server.cc:307] RPC server started. Bound to: 127.14.103.3:39881
I20260812 02:32:47.466540 25645 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.103.3:39881 every 8 connection(s)
I20260812 02:32:47.467588 25493 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786501898713699-14748-0/minicluster-data/ts-2/data/info.pb
I20260812 02:32:47.474117 14748 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbep4M4/build/debug/bin/kudu as pid 25493
I20260812 02:32:47.474264 14748 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786501898713699-14748-0/minicluster-data/ts-2/wal/instance
I20260812 02:32:47.474635 14748 external_mini_cluster.cc:1428] Setting key def0346fae6213916f720f6266b555ad
I20260812 02:32:47.479764 25163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:47.469890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.3:52591 (local address 127.14.103.62:40695)
0812 02:32:47.470310 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:47.470315 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:47.471555 (+ 1240us) server_negotiation.cc:407] Connection header received
0812 02:32:47.473073 (+ 1518us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:47.473076 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:47.473145 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:47.473260 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:32:47.474529 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:47.475337 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:47.477240 (+ 1903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:47.477566 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:47.477668 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:47.479007 (+ 1339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:32:47.479010 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:32:47.479012 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:32:47.479028 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:32:47.479051 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:32:47.479060 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:32:47.479159 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0812 02:32:47.479554 (+ 395us) server_negotiation.cc:299] Negotiation successful
0812 02:32:47.479629 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300}
I20260812 02:32:47.480906 25649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:47.470317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:40695 (local address 127.14.103.3:52591)
0812 02:32:47.471378 (+ 1061us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:47.471430 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0812 02:32:47.472808 (+ 1378us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:32:47.473464 (+ 656us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:32:47.473472 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:32:47.473568 (+ 96us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:32:47.474316 (+ 748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:47.474339 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:47.475598 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:47.475602 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:47.477082 (+ 1480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:47.477099 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:47.477782 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:47.477786 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:47.477975 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:47.478667 (+ 692us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 02:32:47.478669 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:32:47.478716 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 02:32:47.478792 (+ 76us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:32:47.479261 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:32:47.479279 (+ 18us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:32:47.479288 (+ 9us) client_negotiation.cc:769] Sending connection context
0812 02:32:47.479505 (+ 217us) client_negotiation.cc:240] Negotiation successful
0812 02:32:47.480001 (+ 496us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":142,"threads_started":1}
I20260812 02:32:47.482193 25647 heartbeater.cc:347] Connected to a master server at 127.14.103.62:40695
I20260812 02:32:47.482509 25647 heartbeater.cc:464] Registering TS with master...
I20260812 02:32:47.483328 25647 heartbeater.cc:511] Master 127.14.103.62:40695 requested a full tablet report, sending...
I20260812 02:32:47.484661 25117 ts_manager.cc:194] Registered new tserver with Master: 98baacd9865647899cad6ac4ba481a09 (127.14.103.3:39881)
I20260812 02:32:47.485328 25117 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.103.3:52591
I20260812 02:32:47.490005 14748 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260812 02:32:47.493786 25163 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:32:47.492425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44504 (local address 127.14.103.62:40695)
0812 02:32:47.492601 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:47.492605 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:47.492758 (+ 153us) server_negotiation.cc:407] Connection header received
0812 02:32:47.492912 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:47.492915 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:47.492966 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:47.493046 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:32:47.493597 (+ 551us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44504: BlockingRecv error: recv got EOF from 127.0.0.1:44504 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
I20260812 02:32:47.494125 14748 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbep4M4/build/debug/bin/kudu with pid 25187
I20260812 02:32:47.502252 14748 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbep4M4/build/debug/bin/kudu with pid 25339
I20260812 02:32:47.509896 14748 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbep4M4/build/debug/bin/kudu with pid 25493
I20260812 02:32:47.518548 14748 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbep4M4/build/debug/bin/kudu with pid 25080
2026-08-12T02:32:47Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (1020 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-08-12T02:32:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T02:32:47Z Disabled control of system clock
I20260812 02:32:47.554761 14748 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbep4M4/build/debug/bin/kudu
/tmp/dist-test-taskbep4M4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786501898713699-14748-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786501898713699-14748-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786501898713699-14748-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786501898713699-14748-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.103.62:36725
--webserver_interface=127.14.103.62
--webserver_port=0
--builtin_ntp_servers=127.14.103.20:38127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.103.62:36725
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260812 02:32:47.675568 25660 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:32:47.675889 25660 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:32:47.675931 25660 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:32:47.680511 25660 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:32:47.680612 25660 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:32:47.680629 25660 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:32:47.680647 25660 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:32:47.680663 25660 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 02:32:47.686088 25660 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.103.20:38127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786501898713699-14748-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786501898713699-14748-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.103.62:36725
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.103.62:36725
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786501898713699-14748-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.103.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25660
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786501898713699-14748-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:32:47.687340 25660 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:32:47.688750 25660 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:32:47.698834 25669 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:32:47.698875 25660 server_base.cc:1034] running on GCE node
W20260812 02:32:47.699069 25666 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:32:47.699111 25665 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:32:47.699779 25660 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:32:47.701099 25660 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:32:47.702323 25660 hybrid_clock.cc:648] HybridClock initialized: now 1786501967702304 us; error 37 us; skew 500 ppm
I20260812 02:32:47.704608 25660 webserver.cc:460] Webserver started at http://127.14.103.62:44937/ using document root <none> and password file <none>
I20260812 02:32:47.705237 25660 fs_manager.cc:359] Metadata directory not provided
I20260812 02:32:47.705334 25660 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:32:47.705583 25660 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:32:47.707828 25660 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786501898713699-14748-0/minicluster-data/master-0/data/instance:
uuid: "142802983b844404af0ca2f74629a2d8"
format_stamp: "Formatted at 2026-08-12 02:32:47 on dist-test-slave-2w3w"
server_key: "0b7601570f1e82d7e9a93073e8602403"
server_key_iv: "d7255e0f129b499f213938e3ff3484f1"
server_key_version: "encryptionkey@0"
I20260812 02:32:47.708425 25660 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786501898713699-14748-0/minicluster-data/master-0/wal/instance:
uuid: "142802983b844404af0ca2f74629a2d8"
format_stamp: "Formatted at 2026-08-12 02:32:47 on dist-test-slave-2w3w"
server_key: "0b7601570f1e82d7e9a93073e8602403"
server_key_iv: "d7255e0f129b499f213938e3ff3484f1"
server_key_version: "encryptionkey@0"
I20260812 02:32:47.712477 25660 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260812 02:32:47.715251 25676 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:32:47.716667 25660 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260812 02:32:47.716847 25660 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786501898713699-14748-0/minicluster-data/master-0/data,/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786501898713699-14748-0/minicluster-data/master-0/wal
uuid: "142802983b844404af0ca2f74629a2d8"
format_stamp: "Formatted at 2026-08-12 02:32:47 on dist-test-slave-2w3w"
server_key: "0b7601570f1e82d7e9a93073e8602403"
server_key_iv: "d7255e0f129b499f213938e3ff3484f1"
server_key_version: "encryptionkey@0"
I20260812 02:32:47.716984 25660 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786501898713699-14748-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786501898713699-14748-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786501898713699-14748-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:32:47.721997 25660 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:32:47.722641 25660 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:32:47.722841 25660 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:32:47.730643 25660 rpc_server.cc:307] RPC server started. Bound to: 127.14.103.62:36725
I20260812 02:32:47.730660 25738 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.103.62:36725 every 8 connection(s)
I20260812 02:32:47.731709 25660 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786501898713699-14748-0/minicluster-data/master-0/data/info.pb
I20260812 02:32:47.732239 14748 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbep4M4/build/debug/bin/kudu as pid 25660
I20260812 02:32:47.732367 14748 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786501898713699-14748-0/minicluster-data/master-0/wal/instance
I20260812 02:32:47.732649 14748 external_mini_cluster.cc:1428] Setting key 215c2b7d2534a8fdc3831a59c24a0e29
I20260812 02:32:47.735697 25739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:32:47.739557 25742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:47.734227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35784 (local address 127.14.103.62:36725)
0812 02:32:47.735593 (+ 1366us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:47.735609 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:47.735650 (+ 41us) server_negotiation.cc:407] Connection header received
0812 02:32:47.736409 (+ 759us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:47.736428 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:47.736787 (+ 359us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:47.737179 (+ 392us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:32:47.738115 (+ 936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:32:47.738120 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:32:47.738133 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:32:47.738144 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:32:47.738203 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:32:47.738220 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:32:47.738389 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0812 02:32:47.738466 (+ 77us) server_negotiation.cc:299] Negotiation successful
0812 02:32:47.738727 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":983,"thread_start_us":768,"threads_started":1}
I20260812 02:32:47.743243 25739 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 142802983b844404af0ca2f74629a2d8: Bootstrap starting.
I20260812 02:32:47.745645 25739 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 142802983b844404af0ca2f74629a2d8: Neither blocks nor log segments found. Creating new log.
I20260812 02:32:47.746444 25739 log.cc:824] T 00000000000000000000000000000000 P 142802983b844404af0ca2f74629a2d8: Log is configured to *not* fsync() on all Append() calls
I20260812 02:32:47.748941 25739 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 142802983b844404af0ca2f74629a2d8: No bootstrap required, opened a new log
I20260812 02:32:47.752708 25739 raft_consensus.cc:348] T 00000000000000000000000000000000 P 142802983b844404af0ca2f74629a2d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "142802983b844404af0ca2f74629a2d8" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 36725 } }
I20260812 02:32:47.752976 25739 raft_consensus.cc:374] T 00000000000000000000000000000000 P 142802983b844404af0ca2f74629a2d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:32:47.753014 25739 raft_consensus.cc:729] T 00000000000000000000000000000000 P 142802983b844404af0ca2f74629a2d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 142802983b844404af0ca2f74629a2d8, State: Initialized, Role: FOLLOWER
I20260812 02:32:47.753460 25739 consensus_queue.cc:260] T 00000000000000000000000000000000 P 142802983b844404af0ca2f74629a2d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "142802983b844404af0ca2f74629a2d8" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 36725 } }
I20260812 02:32:47.753587 25739 raft_consensus.cc:388] T 00000000000000000000000000000000 P 142802983b844404af0ca2f74629a2d8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 02:32:47.753628 25739 raft_consensus.cc:482] T 00000000000000000000000000000000 P 142802983b844404af0ca2f74629a2d8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 02:32:47.753703 25739 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 142802983b844404af0ca2f74629a2d8 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:32:47.754930 25739 raft_consensus.cc:504] T 00000000000000000000000000000000 P 142802983b844404af0ca2f74629a2d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "142802983b844404af0ca2f74629a2d8" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 36725 } }
I20260812 02:32:47.755398 25739 leader_election.cc:302] T 00000000000000000000000000000000 P 142802983b844404af0ca2f74629a2d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 142802983b844404af0ca2f74629a2d8; no voters:
I20260812 02:32:47.755754 25739 leader_election.cc:288] T 00000000000000000000000000000000 P 142802983b844404af0ca2f74629a2d8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 02:32:47.756027 25744 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 142802983b844404af0ca2f74629a2d8 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:32:47.756354 25744 raft_consensus.cc:686] T 00000000000000000000000000000000 P 142802983b844404af0ca2f74629a2d8 [term 1 LEADER]: Becoming Leader. State: Replica: 142802983b844404af0ca2f74629a2d8, State: Running, Role: LEADER
I20260812 02:32:47.756711 25744 consensus_queue.cc:237] T 00000000000000000000000000000000 P 142802983b844404af0ca2f74629a2d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "142802983b844404af0ca2f74629a2d8" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 36725 } }
I20260812 02:32:47.757222 25739 sys_catalog.cc:564] T 00000000000000000000000000000000 P 142802983b844404af0ca2f74629a2d8 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:32:47.758699 25746 sys_catalog.cc:455] T 00000000000000000000000000000000 P 142802983b844404af0ca2f74629a2d8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 142802983b844404af0ca2f74629a2d8. Latest consensus state: current_term: 1 leader_uuid: "142802983b844404af0ca2f74629a2d8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "142802983b844404af0ca2f74629a2d8" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 36725 } } }
I20260812 02:32:47.758838 25746 sys_catalog.cc:458] T 00000000000000000000000000000000 P 142802983b844404af0ca2f74629a2d8 [sys.catalog]: This master's current role is: LEADER
I20260812 02:32:47.759130 25745 sys_catalog.cc:455] T 00000000000000000000000000000000 P 142802983b844404af0ca2f74629a2d8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "142802983b844404af0ca2f74629a2d8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "142802983b844404af0ca2f74629a2d8" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 36725 } } }
I20260812 02:32:47.759235 25745 sys_catalog.cc:458] T 00000000000000000000000000000000 P 142802983b844404af0ca2f74629a2d8 [sys.catalog]: This master's current role is: LEADER
W20260812 02:32:47.762840 25763 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 142802983b844404af0ca2f74629a2d8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 02:32:47.763022 25763 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 02:32:47.763120 25755 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 02:32:47.763916 25755 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 02:32:47.769861 25755 catalog_manager.cc:1347] Generated new cluster ID: b91b7ac0397a4bae8cfb0e1967d5133d
I20260812 02:32:47.769948 25755 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 02:32:47.783242 25755 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 02:32:47.783639 25755 catalog_manager.cc:1504] Loading token signing keys...
I20260812 02:32:47.795076 25755 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 142802983b844404af0ca2f74629a2d8: Generated new TSK 0
I20260812 02:32:47.796002 25755 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 02:32:47.808634 14748 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbep4M4/build/debug/bin/kudu
/tmp/dist-test-taskbep4M4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786501898713699-14748-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786501898713699-14748-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786501898713699-14748-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786501898713699-14748-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.103.1:0
--local_ip_for_outbound_sockets=127.14.103.1
--webserver_interface=127.14.103.1
--webserver_port=0
--tserver_master_addrs=127.14.103.62:36725
--builtin_ntp_servers=127.14.103.20:38127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260812 02:32:47.951735 25767 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:32:47.952039 25767 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:32:47.952081 25767 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:32:47.956353 25767 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:32:47.956430 25767 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:32:47.956491 25767 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.103.1
I20260812 02:32:47.962249 25767 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.103.20:38127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786501898713699-14748-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786501898713699-14748-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.103.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786501898713699-14748-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.103.1
--webserver_port=0
--tserver_master_addrs=127.14.103.62:36725
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25767
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.103.1
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786501898713699-14748-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:32:47.963589 25767 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:32:47.965142 25767 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:32:47.975757 25774 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:32:47.975800 25773 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:32:47.975818 25776 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:32:47.976449 25767 server_base.cc:1034] running on GCE node
I20260812 02:32:47.976984 25767 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:32:47.977818 25767 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:32:47.979063 25767 hybrid_clock.cc:648] HybridClock initialized: now 1786501967979048 us; error 39 us; skew 500 ppm
I20260812 02:32:47.981236 25767 webserver.cc:460] Webserver started at http://127.14.103.1:44627/ using document root <none> and password file <none>
I20260812 02:32:47.981931 25767 fs_manager.cc:359] Metadata directory not provided
I20260812 02:32:47.982168 25767 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:32:47.982450 25767 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:32:47.984804 25767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786501898713699-14748-0/minicluster-data/ts-0/data/instance:
uuid: "1602c4070dc14ea1b17a47b4eb50b496"
format_stamp: "Formatted at 2026-08-12 02:32:47 on dist-test-slave-2w3w"
server_key: "f3f3bbd206b335097cce21c3477f3858"
server_key_iv: "4b873a7f53d191c76833a48747af60a9"
server_key_version: "encryptionkey@0"
I20260812 02:32:47.985428 25767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786501898713699-14748-0/minicluster-data/ts-0/wal/instance:
uuid: "1602c4070dc14ea1b17a47b4eb50b496"
format_stamp: "Formatted at 2026-08-12 02:32:47 on dist-test-slave-2w3w"
server_key: "f3f3bbd206b335097cce21c3477f3858"
server_key_iv: "4b873a7f53d191c76833a48747af60a9"
server_key_version: "encryptionkey@0"
I20260812 02:32:47.989952 25767 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260812 02:32:47.993245 25785 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:32:47.994784 25767 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260812 02:32:47.995043 25767 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786501898713699-14748-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786501898713699-14748-0/minicluster-data/ts-0/wal
uuid: "1602c4070dc14ea1b17a47b4eb50b496"
format_stamp: "Formatted at 2026-08-12 02:32:47 on dist-test-slave-2w3w"
server_key: "f3f3bbd206b335097cce21c3477f3858"
server_key_iv: "4b873a7f53d191c76833a48747af60a9"
server_key_version: "encryptionkey@0"
I20260812 02:32:47.995200 25767 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786501898713699-14748-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786501898713699-14748-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786501898713699-14748-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:32:48.000875 25767 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:32:48.001780 25767 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:32:48.002067 25767 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:32:48.003261 25767 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:32:48.004772 25767 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:32:48.004859 25767 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:32:48.005175 25767 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:32:48.005251 25767 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:32:48.016263 25767 rpc_server.cc:307] RPC server started. Bound to: 127.14.103.1:35057
I20260812 02:32:48.016280 25917 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.103.1:35057 every 8 connection(s)
I20260812 02:32:48.017267 25767 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786501898713699-14748-0/minicluster-data/ts-0/data/info.pb
I20260812 02:32:48.018792 14748 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbep4M4/build/debug/bin/kudu as pid 25767
I20260812 02:32:48.018988 14748 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786501898713699-14748-0/minicluster-data/ts-0/wal/instance
I20260812 02:32:48.019418 14748 external_mini_cluster.cc:1428] Setting key d9d991f82c991f2356e40be96d551272
I20260812 02:32:48.022023 14748 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbep4M4/build/debug/bin/kudu
/tmp/dist-test-taskbep4M4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786501898713699-14748-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786501898713699-14748-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786501898713699-14748-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786501898713699-14748-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.103.2:0
--local_ip_for_outbound_sockets=127.14.103.2
--webserver_interface=127.14.103.2
--webserver_port=0
--tserver_master_addrs=127.14.103.62:36725
--builtin_ntp_servers=127.14.103.20:38127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260812 02:32:48.025732 25742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:48.019526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.1:37915 (local address 127.14.103.62:36725)
0812 02:32:48.019681 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:48.019684 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:48.021112 (+ 1428us) server_negotiation.cc:407] Connection header received
0812 02:32:48.023205 (+ 2093us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:48.023208 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:48.023249 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:48.023332 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:32:48.025014 (+ 1682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:32:48.025020 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:32:48.025025 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:32:48.025044 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:32:48.025080 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:32:48.025092 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:32:48.025222 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0812 02:32:48.025478 (+ 256us) server_negotiation.cc:299] Negotiation successful
0812 02:32:48.025534 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260812 02:32:48.026883 25920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:48.020029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:36725 (local address 127.14.103.1:37915)
0812 02:32:48.020946 (+ 917us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:48.021005 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0812 02:32:48.022886 (+ 1881us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:32:48.023568 (+ 682us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:32:48.023580 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:32:48.023711 (+ 131us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:32:48.024687 (+ 976us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 02:32:48.024689 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:32:48.024737 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 02:32:48.024817 (+ 80us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:32:48.025271 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:32:48.025281 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:32:48.025286 (+ 5us) client_negotiation.cc:769] Sending connection context
0812 02:32:48.025487 (+ 201us) client_negotiation.cc:240] Negotiation successful
0812 02:32:48.025709 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":216,"threads_started":1}
I20260812 02:32:48.028420 25918 heartbeater.cc:347] Connected to a master server at 127.14.103.62:36725
I20260812 02:32:48.028671 25918 heartbeater.cc:464] Registering TS with master...
I20260812 02:32:48.029134 25918 heartbeater.cc:511] Master 127.14.103.62:36725 requested a full tablet report, sending...
I20260812 02:32:48.030766 25695 ts_manager.cc:194] Registered new tserver with Master: 1602c4070dc14ea1b17a47b4eb50b496 (127.14.103.1:35057)
W20260812 02:32:48.141319 25922 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:32:48.141619 25922 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:32:48.141664 25922 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:32:48.146057 25922 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:32:48.146142 25922 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:32:48.146204 25922 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.103.2
I20260812 02:32:48.151338 25922 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.103.20:38127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786501898713699-14748-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786501898713699-14748-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.103.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786501898713699-14748-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.103.2
--webserver_port=0
--tserver_master_addrs=127.14.103.62:36725
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25922
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.103.2
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786501898713699-14748-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:32:48.152539 25922 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:32:48.154009 25922 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:32:48.164814 25927 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:32:48.164825 25930 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:32:48.164805 25928 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:32:48.165272 25922 server_base.cc:1034] running on GCE node
I20260812 02:32:48.165767 25922 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:32:48.166479 25922 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:32:48.167721 25922 hybrid_clock.cc:648] HybridClock initialized: now 1786501968167685 us; error 56 us; skew 500 ppm
I20260812 02:32:48.173504 25922 webserver.cc:460] Webserver started at http://127.14.103.2:38525/ using document root <none> and password file <none>
I20260812 02:32:48.174214 25922 fs_manager.cc:359] Metadata directory not provided
I20260812 02:32:48.174326 25922 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:32:48.174584 25922 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:32:48.176929 25922 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786501898713699-14748-0/minicluster-data/ts-1/data/instance:
uuid: "4de49d54f5d744f597b54ef34f4eca44"
format_stamp: "Formatted at 2026-08-12 02:32:48 on dist-test-slave-2w3w"
server_key: "8586d9651a6353a35223d632030de076"
server_key_iv: "493520a2cc77f25f68ea8a51a5574127"
server_key_version: "encryptionkey@0"
I20260812 02:32:48.177560 25922 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786501898713699-14748-0/minicluster-data/ts-1/wal/instance:
uuid: "4de49d54f5d744f597b54ef34f4eca44"
format_stamp: "Formatted at 2026-08-12 02:32:48 on dist-test-slave-2w3w"
server_key: "8586d9651a6353a35223d632030de076"
server_key_iv: "493520a2cc77f25f68ea8a51a5574127"
server_key_version: "encryptionkey@0"
I20260812 02:32:48.181860 25922 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260812 02:32:48.184926 25937 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:32:48.186416 25922 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260812 02:32:48.186682 25922 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786501898713699-14748-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786501898713699-14748-0/minicluster-data/ts-1/wal
uuid: "4de49d54f5d744f597b54ef34f4eca44"
format_stamp: "Formatted at 2026-08-12 02:32:48 on dist-test-slave-2w3w"
server_key: "8586d9651a6353a35223d632030de076"
server_key_iv: "493520a2cc77f25f68ea8a51a5574127"
server_key_version: "encryptionkey@0"
I20260812 02:32:48.186846 25922 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786501898713699-14748-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786501898713699-14748-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786501898713699-14748-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:32:48.192152 25922 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:32:48.192850 25922 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:32:48.193009 25922 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:32:48.193547 25922 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:32:48.194536 25922 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:32:48.194586 25922 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:32:48.194626 25922 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:32:48.194641 25922 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:32:48.204782 25922 rpc_server.cc:307] RPC server started. Bound to: 127.14.103.2:37525
I20260812 02:32:48.204900 26071 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.103.2:37525 every 8 connection(s)
I20260812 02:32:48.205976 25922 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786501898713699-14748-0/minicluster-data/ts-1/data/info.pb
I20260812 02:32:48.211310 14748 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbep4M4/build/debug/bin/kudu as pid 25922
I20260812 02:32:48.211503 14748 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786501898713699-14748-0/minicluster-data/ts-1/wal/instance
I20260812 02:32:48.211835 14748 external_mini_cluster.cc:1428] Setting key afacf34f304979897809fc182927ca5c
I20260812 02:32:48.214347 14748 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbep4M4/build/debug/bin/kudu
/tmp/dist-test-taskbep4M4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786501898713699-14748-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786501898713699-14748-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786501898713699-14748-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786501898713699-14748-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.103.3:0
--local_ip_for_outbound_sockets=127.14.103.3
--webserver_interface=127.14.103.3
--webserver_port=0
--tserver_master_addrs=127.14.103.62:36725
--builtin_ntp_servers=127.14.103.20:38127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260812 02:32:48.216677 25742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:48.208114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.2:58207 (local address 127.14.103.62:36725)
0812 02:32:48.208292 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:48.208296 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:48.209433 (+ 1137us) server_negotiation.cc:407] Connection header received
0812 02:32:48.210816 (+ 1383us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:48.210820 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:48.210871 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:48.210986 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:32:48.216151 (+ 5165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:32:48.216155 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:32:48.216158 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:32:48.216173 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:32:48.216198 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:32:48.216215 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:32:48.216321 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0812 02:32:48.216490 (+ 169us) server_negotiation.cc:299] Negotiation successful
0812 02:32:48.216539 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260812 02:32:48.217664 26074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:48.208508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:36725 (local address 127.14.103.2:58207)
0812 02:32:48.209245 (+ 737us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:48.209290 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0812 02:32:48.210578 (+ 1288us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:32:48.211195 (+ 617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:32:48.211202 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:32:48.211299 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:32:48.215785 (+ 4486us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 02:32:48.215787 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:32:48.215856 (+ 69us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 02:32:48.215959 (+ 103us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:32:48.216307 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:32:48.216318 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:32:48.216334 (+ 16us) client_negotiation.cc:769] Sending connection context
0812 02:32:48.216550 (+ 216us) client_negotiation.cc:240] Negotiation successful
0812 02:32:48.216851 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":133,"threads_started":1}
I20260812 02:32:48.219164 26072 heartbeater.cc:347] Connected to a master server at 127.14.103.62:36725
I20260812 02:32:48.219465 26072 heartbeater.cc:464] Registering TS with master...
I20260812 02:32:48.219955 26072 heartbeater.cc:511] Master 127.14.103.62:36725 requested a full tablet report, sending...
I20260812 02:32:48.223354 25695 ts_manager.cc:194] Registered new tserver with Master: 4de49d54f5d744f597b54ef34f4eca44 (127.14.103.2:37525)
W20260812 02:32:48.341331 26075 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:32:48.341668 26075 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:32:48.341749 26075 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:32:48.346187 26075 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:32:48.346318 26075 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:32:48.346427 26075 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.103.3
I20260812 02:32:48.351963 26075 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.103.20:38127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786501898713699-14748-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786501898713699-14748-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.103.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786501898713699-14748-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.103.3
--webserver_port=0
--tserver_master_addrs=127.14.103.62:36725
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26075
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.103.3
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786501898713699-14748-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:32:48.353181 26075 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:32:48.354622 26075 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:32:48.365461 26082 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:32:48.365485 26080 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:32:48.365461 26086 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:32:48.365795 26075 server_base.cc:1034] running on GCE node
I20260812 02:32:48.366370 26075 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:32:48.367127 26075 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:32:48.368373 26075 hybrid_clock.cc:648] HybridClock initialized: now 1786501968368344 us; error 58 us; skew 500 ppm
I20260812 02:32:48.370648 26075 webserver.cc:460] Webserver started at http://127.14.103.3:35329/ using document root <none> and password file <none>
I20260812 02:32:48.371433 26075 fs_manager.cc:359] Metadata directory not provided
I20260812 02:32:48.371539 26075 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:32:48.371795 26075 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:32:48.374145 26075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786501898713699-14748-0/minicluster-data/ts-2/data/instance:
uuid: "e1e70e60f24b4763a4b9650d5c96876a"
format_stamp: "Formatted at 2026-08-12 02:32:48 on dist-test-slave-2w3w"
server_key: "13996badaf3b17e030933e8a2ceb4a0b"
server_key_iv: "7b761b6095b13897d67ec36178784185"
server_key_version: "encryptionkey@0"
I20260812 02:32:48.374828 26075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786501898713699-14748-0/minicluster-data/ts-2/wal/instance:
uuid: "e1e70e60f24b4763a4b9650d5c96876a"
format_stamp: "Formatted at 2026-08-12 02:32:48 on dist-test-slave-2w3w"
server_key: "13996badaf3b17e030933e8a2ceb4a0b"
server_key_iv: "7b761b6095b13897d67ec36178784185"
server_key_version: "encryptionkey@0"
I20260812 02:32:48.379062 26075 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260812 02:32:48.381892 26092 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:32:48.383363 26075 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260812 02:32:48.383519 26075 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786501898713699-14748-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786501898713699-14748-0/minicluster-data/ts-2/wal
uuid: "e1e70e60f24b4763a4b9650d5c96876a"
format_stamp: "Formatted at 2026-08-12 02:32:48 on dist-test-slave-2w3w"
server_key: "13996badaf3b17e030933e8a2ceb4a0b"
server_key_iv: "7b761b6095b13897d67ec36178784185"
server_key_version: "encryptionkey@0"
I20260812 02:32:48.383654 26075 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786501898713699-14748-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786501898713699-14748-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786501898713699-14748-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:32:48.388844 26075 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:32:48.389534 26075 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:32:48.389735 26075 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:32:48.390317 26075 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:32:48.391393 26075 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:32:48.391465 26075 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:32:48.391537 26075 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:32:48.391575 26075 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:32:48.401250 26075 rpc_server.cc:307] RPC server started. Bound to: 127.14.103.3:45543
I20260812 02:32:48.401448 26221 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.103.3:45543 every 8 connection(s)
I20260812 02:32:48.402316 26075 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786501898713699-14748-0/minicluster-data/ts-2/data/info.pb
I20260812 02:32:48.403175 14748 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbep4M4/build/debug/bin/kudu as pid 26075
I20260812 02:32:48.403451 14748 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786501898713699-14748-0/minicluster-data/ts-2/wal/instance
I20260812 02:32:48.403797 14748 external_mini_cluster.cc:1428] Setting key 39b3418785113dca1ab914a006c16021
I20260812 02:32:48.410095 25742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:48.405269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.3:49415 (local address 127.14.103.62:36725)
0812 02:32:48.405420 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:48.405425 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:48.406928 (+ 1503us) server_negotiation.cc:407] Connection header received
0812 02:32:48.408277 (+ 1349us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:48.408280 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:48.408332 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:48.408432 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:32:48.409650 (+ 1218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:32:48.409654 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:32:48.409656 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:32:48.409669 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:32:48.409691 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:32:48.409700 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:32:48.409787 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0812 02:32:48.409936 (+ 149us) server_negotiation.cc:299] Negotiation successful
0812 02:32:48.409979 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260812 02:32:48.411631 26224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:48.405877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:36725 (local address 127.14.103.3:49415)
0812 02:32:48.406746 (+ 869us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:48.406802 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0812 02:32:48.408053 (+ 1251us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:32:48.408601 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:32:48.408608 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:32:48.408705 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:32:48.409364 (+ 659us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 02:32:48.409366 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:32:48.409416 (+ 50us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 02:32:48.409490 (+ 74us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:32:48.409785 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:32:48.409791 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:32:48.409795 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 02:32:48.409978 (+ 183us) client_negotiation.cc:240] Negotiation successful
0812 02:32:48.410404 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":164,"threads_started":1}
I20260812 02:32:48.413017 26222 heartbeater.cc:347] Connected to a master server at 127.14.103.62:36725
I20260812 02:32:48.413255 26222 heartbeater.cc:464] Registering TS with master...
I20260812 02:32:48.413676 26222 heartbeater.cc:511] Master 127.14.103.62:36725 requested a full tablet report, sending...
I20260812 02:32:48.414848 25695 ts_manager.cc:194] Registered new tserver with Master: e1e70e60f24b4763a4b9650d5c96876a (127.14.103.3:45543)
I20260812 02:32:48.418480 14748 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260812 02:32:48.421977 25742 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:32:48.420874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51704 (local address 127.14.103.62:36725)
0812 02:32:48.421059 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:48.421064 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:48.421250 (+ 186us) server_negotiation.cc:407] Connection header received
0812 02:32:48.421468 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:48.421471 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:48.421508 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:48.421594 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:32:48.421853 (+ 259us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51704: BlockingRecv error: recv got EOF from 127.0.0.1:51704 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
I20260812 02:32:48.422554 14748 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbep4M4/build/debug/bin/kudu with pid 25767
I20260812 02:32:48.440140 14748 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbep4M4/build/debug/bin/kudu with pid 25922
I20260812 02:32:48.450742 14748 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbep4M4/build/debug/bin/kudu with pid 26075
I20260812 02:32:48.458345 14748 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbep4M4/build/debug/bin/kudu with pid 25660
2026-08-12T02:32:48Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (938 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501898713699-14748-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 02:32:48 dist-test-slave-2w3w krb5kdc[26232](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 02:32:48 dist-test-slave-2w3w krb5kdc[26232](info): set up 2 sockets
Aug 12 02:32:48 dist-test-slave-2w3w krb5kdc[26232](info): commencing operation
krb5kdc: starting...
W20260812 02:32:49.616572 14748 mini_kdc.cc:121] Time spent starting KDC: real 1.133s user 0.001s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501898713699-14748-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501898713699-14748-0/krb5kdc/test-user.keytab.
Aug 12 02:32:49 dist-test-slave-2w3w krb5kdc[26232](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501969, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T02:32:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T02:32:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.103.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.103.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.103.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.62.keytab.
Entry for principal kudu/127.14.103.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.62.keytab.
I20260812 02:32:49.772446 14748 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbep4M4/build/debug/bin/kudu
/tmp/dist-test-taskbep4M4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501898713699-14748-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501898713699-14748-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501898713699-14748-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501898713699-14748-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.103.62:42157
--webserver_interface=127.14.103.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.62.keytab
--principal=kudu/127.14.103.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.103.20:36113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.103.62:42157
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501898713699-14748-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501898713699-14748-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501898713699-14748-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:32:49.897822 26247 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:32:49.898216 26247 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:32:49.898313 26247 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:32:49.902555 26247 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:32:49.902678 26247 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:32:49.902719 26247 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:32:49.902772 26247 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:32:49.902809 26247 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 02:32:49.908557 26247 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.103.20:36113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501898713699-14748-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501898713699-14748-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.103.62:42157
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.62.keytab
--principal=kudu/127.14.103.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.103.62:42157
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501898713699-14748-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.103.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26247
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501898713699-14748-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:32:49.909797 26247 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:32:49.911020 26247 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:32:49.920445 26252 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:32:49.920436 26253 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:32:49.920517 26247 server_base.cc:1034] running on GCE node
W20260812 02:32:49.920441 26255 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:32:49.921563 26247 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:32:49.922760 26247 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:32:49.923993 26247 hybrid_clock.cc:648] HybridClock initialized: now 1786501969923975 us; error 40 us; skew 500 ppm
Aug 12 02:32:49 dist-test-slave-2w3w krb5kdc[26232](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501969, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:32:49.927390 26247 init.cc:377] Logged in from keytab as kudu/127.14.103.62@KRBTEST.COM (short username kudu)
I20260812 02:32:49.928517 26247 webserver.cc:460] Webserver started at http://127.14.103.62:39715/ using document root <none> and password file <none>
I20260812 02:32:49.929293 26247 fs_manager.cc:359] Metadata directory not provided
I20260812 02:32:49.929459 26247 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:32:49.929740 26247 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:32:49.932165 26247 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501898713699-14748-0/minicluster-data/master-0/data/instance:
uuid: "9dbd1a3ca0254e8696c3ab074724f460"
format_stamp: "Formatted at 2026-08-12 02:32:49 on dist-test-slave-2w3w"
server_key: "5aa6c89a9f7156ca5ad399626d713586"
server_key_iv: "bf50d0466f328494956bd0286848fcbe"
server_key_version: "encryptionkey@0"
I20260812 02:32:49.932845 26247 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501898713699-14748-0/minicluster-data/master-0/wal/instance:
uuid: "9dbd1a3ca0254e8696c3ab074724f460"
format_stamp: "Formatted at 2026-08-12 02:32:49 on dist-test-slave-2w3w"
server_key: "5aa6c89a9f7156ca5ad399626d713586"
server_key_iv: "bf50d0466f328494956bd0286848fcbe"
server_key_version: "encryptionkey@0"
I20260812 02:32:49.937654 26247 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260812 02:32:49.940850 26263 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:32:49.942384 26247 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260812 02:32:49.942536 26247 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501898713699-14748-0/minicluster-data/master-0/data,/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501898713699-14748-0/minicluster-data/master-0/wal
uuid: "9dbd1a3ca0254e8696c3ab074724f460"
format_stamp: "Formatted at 2026-08-12 02:32:49 on dist-test-slave-2w3w"
server_key: "5aa6c89a9f7156ca5ad399626d713586"
server_key_iv: "bf50d0466f328494956bd0286848fcbe"
server_key_version: "encryptionkey@0"
I20260812 02:32:49.942672 26247 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501898713699-14748-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501898713699-14748-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501898713699-14748-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:32:49.961488 26247 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:32:49.965307 26247 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:32:49.965651 26247 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:32:49.974671 26247 rpc_server.cc:307] RPC server started. Bound to: 127.14.103.62:42157
I20260812 02:32:49.975138 26322 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.103.62:42157 every 8 connection(s)
I20260812 02:32:49.975844 26247 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501898713699-14748-0/minicluster-data/master-0/data/info.pb
I20260812 02:32:49.978935 26323 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:32:49.981696 14748 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbep4M4/build/debug/bin/kudu as pid 26247
I20260812 02:32:49.982002 14748 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501898713699-14748-0/minicluster-data/master-0/wal/instance
I20260812 02:32:49.982326 14748 external_mini_cluster.cc:1428] Setting key 708ce2b0b55b7ce070f9b348475b1fac
I20260812 02:32:49.985853 26323 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9dbd1a3ca0254e8696c3ab074724f460: Bootstrap starting.
I20260812 02:32:49.988874 26323 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9dbd1a3ca0254e8696c3ab074724f460: Neither blocks nor log segments found. Creating new log.
I20260812 02:32:49.989765 26323 log.cc:824] T 00000000000000000000000000000000 P 9dbd1a3ca0254e8696c3ab074724f460: Log is configured to *not* fsync() on all Append() calls
I20260812 02:32:49.992452 26323 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9dbd1a3ca0254e8696c3ab074724f460: No bootstrap required, opened a new log
Aug 12 02:32:49 dist-test-slave-2w3w krb5kdc[26232](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501969, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.103.62@KRBTEST.COM
I20260812 02:32:49.996450 26323 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9dbd1a3ca0254e8696c3ab074724f460 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dbd1a3ca0254e8696c3ab074724f460" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 42157 } }
I20260812 02:32:49.996816 26323 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9dbd1a3ca0254e8696c3ab074724f460 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:32:49.996873 26323 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9dbd1a3ca0254e8696c3ab074724f460 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dbd1a3ca0254e8696c3ab074724f460, State: Initialized, Role: FOLLOWER
I20260812 02:32:49.997416 26323 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9dbd1a3ca0254e8696c3ab074724f460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dbd1a3ca0254e8696c3ab074724f460" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 42157 } }
I20260812 02:32:49.997548 26323 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9dbd1a3ca0254e8696c3ab074724f460 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 02:32:49.997638 26323 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9dbd1a3ca0254e8696c3ab074724f460 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 02:32:49.997751 26323 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9dbd1a3ca0254e8696c3ab074724f460 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:32:49.998868 26323 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9dbd1a3ca0254e8696c3ab074724f460 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dbd1a3ca0254e8696c3ab074724f460" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 42157 } }
I20260812 02:32:49.999251 26323 leader_election.cc:302] T 00000000000000000000000000000000 P 9dbd1a3ca0254e8696c3ab074724f460 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9dbd1a3ca0254e8696c3ab074724f460; no voters:
I20260812 02:32:49.999596 26323 leader_election.cc:288] T 00000000000000000000000000000000 P 9dbd1a3ca0254e8696c3ab074724f460 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 02:32:49.999722 26329 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9dbd1a3ca0254e8696c3ab074724f460 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:32:49.999960 26329 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9dbd1a3ca0254e8696c3ab074724f460 [term 1 LEADER]: Becoming Leader. State: Replica: 9dbd1a3ca0254e8696c3ab074724f460, State: Running, Role: LEADER
I20260812 02:32:49.999979 26326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:49.984038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51298 (local address 127.14.103.62:42157)
0812 02:32:49.984560 (+ 522us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:49.984581 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:49.984629 (+ 48us) server_negotiation.cc:407] Connection header received
0812 02:32:49.985488 (+ 859us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:49.985513 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:49.985967 (+ 454us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:49.986356 (+ 389us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:32:49.987179 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:49.988681 (+ 1502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:49.989669 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:49.990414 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:49.990545 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:49.993662 (+ 3117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:32:49.993667 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:32:49.993681 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:32:49.993710 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:32:49.996126 (+ 2416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:49.996754 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:49.996759 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:49.996764 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:49.997028 (+ 264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:49.997417 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:49.997420 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:49.997422 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:49.997993 (+ 571us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:32:49.998220 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0812 02:32:49.998552 (+ 332us) server_negotiation.cc:299] Negotiation successful
0812 02:32:49.998883 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":119,"threads_started":1}
I20260812 02:32:50.000442 26329 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9dbd1a3ca0254e8696c3ab074724f460 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dbd1a3ca0254e8696c3ab074724f460" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 42157 } }
I20260812 02:32:50.000977 26323 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9dbd1a3ca0254e8696c3ab074724f460 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:32:50.002377 26332 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9dbd1a3ca0254e8696c3ab074724f460 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9dbd1a3ca0254e8696c3ab074724f460. Latest consensus state: current_term: 1 leader_uuid: "9dbd1a3ca0254e8696c3ab074724f460" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dbd1a3ca0254e8696c3ab074724f460" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 42157 } } }
I20260812 02:32:50.002532 26332 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9dbd1a3ca0254e8696c3ab074724f460 [sys.catalog]: This master's current role is: LEADER
I20260812 02:32:50.002388 26330 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9dbd1a3ca0254e8696c3ab074724f460 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9dbd1a3ca0254e8696c3ab074724f460" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dbd1a3ca0254e8696c3ab074724f460" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 42157 } } }
I20260812 02:32:50.002616 26330 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9dbd1a3ca0254e8696c3ab074724f460 [sys.catalog]: This master's current role is: LEADER
W20260812 02:32:50.006492 26352 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9dbd1a3ca0254e8696c3ab074724f460: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 02:32:50.006587 26352 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 02:32:50.006682 26344 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 02:32:50.007531 26344 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 02:32:50.013744 26344 catalog_manager.cc:1347] Generated new cluster ID: 77032b1f154040a7969795ba57a5684b
I20260812 02:32:50.013849 26344 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 02:32:50.024017 26344 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 02:32:50.025229 26344 catalog_manager.cc:1504] Loading token signing keys...
I20260812 02:32:50.034830 26344 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9dbd1a3ca0254e8696c3ab074724f460: Generated new TSK 0
I20260812 02:32:50.035864 26344 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.103.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.103.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.103.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.1.keytab.
Entry for principal kudu/127.14.103.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.1.keytab.
I20260812 02:32:50.076712 14748 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbep4M4/build/debug/bin/kudu
/tmp/dist-test-taskbep4M4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501898713699-14748-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501898713699-14748-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501898713699-14748-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501898713699-14748-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.103.1:0
--local_ip_for_outbound_sockets=127.14.103.1
--webserver_interface=127.14.103.1
--webserver_port=0
--tserver_master_addrs=127.14.103.62:42157
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.1.keytab
--principal=kudu/127.14.103.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.103.20:36113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501898713699-14748-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501898713699-14748-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501898713699-14748-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:32:50.201325 26358 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:32:50.201649 26358 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:32:50.201697 26358 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:32:50.206449 26358 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:32:50.206549 26358 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:32:50.206614 26358 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.103.1
I20260812 02:32:50.212611 26358 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.103.20:36113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501898713699-14748-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501898713699-14748-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.1.keytab
--principal=kudu/127.14.103.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.103.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501898713699-14748-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.103.1
--webserver_port=0
--tserver_master_addrs=127.14.103.62:42157
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26358
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.103.1
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501898713699-14748-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:32:50.213821 26358 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:32:50.214999 26358 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:32:50.225740 26367 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:32:50.225740 26364 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:32:50.225818 26358 server_base.cc:1034] running on GCE node
W20260812 02:32:50.225740 26365 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:32:50.226588 26358 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:32:50.227337 26358 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:32:50.228540 26358 hybrid_clock.cc:648] HybridClock initialized: now 1786501970228521 us; error 34 us; skew 500 ppm
Aug 12 02:32:50 dist-test-slave-2w3w krb5kdc[26232](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501970, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:32:50.231647 26358 init.cc:377] Logged in from keytab as kudu/127.14.103.1@KRBTEST.COM (short username kudu)
I20260812 02:32:50.232623 26358 webserver.cc:460] Webserver started at http://127.14.103.1:46343/ using document root <none> and password file <none>
I20260812 02:32:50.233323 26358 fs_manager.cc:359] Metadata directory not provided
I20260812 02:32:50.233399 26358 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:32:50.233593 26358 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:32:50.235850 26358 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501898713699-14748-0/minicluster-data/ts-0/data/instance:
uuid: "19573b79ad2e48cba2e8fb0885ce88f9"
format_stamp: "Formatted at 2026-08-12 02:32:50 on dist-test-slave-2w3w"
server_key: "65880df20dc206e4b140f446dedb0040"
server_key_iv: "e7507a45694309f2705f14569f6f2e2d"
server_key_version: "encryptionkey@0"
I20260812 02:32:50.236402 26358 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501898713699-14748-0/minicluster-data/ts-0/wal/instance:
uuid: "19573b79ad2e48cba2e8fb0885ce88f9"
format_stamp: "Formatted at 2026-08-12 02:32:50 on dist-test-slave-2w3w"
server_key: "65880df20dc206e4b140f446dedb0040"
server_key_iv: "e7507a45694309f2705f14569f6f2e2d"
server_key_version: "encryptionkey@0"
I20260812 02:32:50.240507 26358 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260812 02:32:50.243511 26374 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:32:50.245059 26358 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 02:32:50.245225 26358 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501898713699-14748-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501898713699-14748-0/minicluster-data/ts-0/wal
uuid: "19573b79ad2e48cba2e8fb0885ce88f9"
format_stamp: "Formatted at 2026-08-12 02:32:50 on dist-test-slave-2w3w"
server_key: "65880df20dc206e4b140f446dedb0040"
server_key_iv: "e7507a45694309f2705f14569f6f2e2d"
server_key_version: "encryptionkey@0"
I20260812 02:32:50.245416 26358 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501898713699-14748-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501898713699-14748-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501898713699-14748-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:32:50.275537 26358 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:32:50.279112 26358 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:32:50.279356 26358 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:32:50.279999 26358 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:32:50.280876 26358 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:32:50.280925 26358 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:32:50.280997 26358 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:32:50.281028 26358 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:32:50.291767 26358 rpc_server.cc:307] RPC server started. Bound to: 127.14.103.1:34111
I20260812 02:32:50.291788 26506 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.103.1:34111 every 8 connection(s)
I20260812 02:32:50.292762 26358 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501898713699-14748-0/minicluster-data/ts-0/data/info.pb
I20260812 02:32:50.295873 14748 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbep4M4/build/debug/bin/kudu as pid 26358
I20260812 02:32:50.296023 14748 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501898713699-14748-0/minicluster-data/ts-0/wal/instance
I20260812 02:32:50.296379 14748 external_mini_cluster.cc:1428] Setting key 4fa227d827e82cce9b6ade6cf4f12a6a
Aug 12 02:32:50 dist-test-slave-2w3w krb5kdc[26232](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501970, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.1@KRBTEST.COM for kudu/127.14.103.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.14.103.2@KRBTEST.COM; defaulting to no policy
I20260812 02:32:50.310619 26326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:50.294800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.1:52327 (local address 127.14.103.62:42157)
0812 02:32:50.294950 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:50.294954 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:50.296042 (+ 1088us) server_negotiation.cc:407] Connection header received
0812 02:32:50.297903 (+ 1861us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:50.297907 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:50.297963 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:50.298068 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:32:50.299888 (+ 1820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:50.300504 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:50.302404 (+ 1900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:50.302864 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:50.303027 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:50.306246 (+ 3219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:32:50.306251 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:32:50.306255 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:32:50.306291 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:32:50.308213 (+ 1922us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:50.308933 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:50.308937 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:50.308938 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:50.309026 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:50.309353 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:50.309356 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:50.309357 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:50.309643 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:32:50.309742 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0812 02:32:50.310335 (+ 593us) server_negotiation.cc:299] Negotiation successful
0812 02:32:50.310487 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260812 02:32:50.311597 26510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:50.295233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:42157 (local address 127.14.103.1:52327)
0812 02:32:50.295846 (+ 613us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:50.295914 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0812 02:32:50.297619 (+ 1705us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:32:50.298292 (+ 673us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:32:50.298302 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:32:50.298880 (+ 578us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:32:50.299672 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:50.299695 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:50.300651 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:50.300659 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:50.302226 (+ 1567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:50.302238 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:50.303082 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:50.303085 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:50.303247 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:50.303978 (+ 731us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:32:50.303996 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:32:50.306008 (+ 2012us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:32:50.308367 (+ 2359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:32:50.308372 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:32:50.308409 (+ 37us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:32:50.308798 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:32:50.309112 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:32:50.309114 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:32:50.309116 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:32:50.309260 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:32:50.309772 (+ 512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:32:50.309780 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:32:50.310087 (+ 307us) client_negotiation.cc:769] Sending connection context
0812 02:32:50.310257 (+ 170us) client_negotiation.cc:240] Negotiation successful
0812 02:32:50.310551 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":116,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.103.2@KRBTEST.COM" created.
I20260812 02:32:50.313158 26507 heartbeater.cc:347] Connected to a master server at 127.14.103.62:42157
I20260812 02:32:50.313445 26507 heartbeater.cc:464] Registering TS with master...
I20260812 02:32:50.313997 26507 heartbeater.cc:511] Master 127.14.103.62:42157 requested a full tablet report, sending...
I20260812 02:32:50.315762 26283 ts_manager.cc:194] Registered new tserver with Master: 19573b79ad2e48cba2e8fb0885ce88f9 (127.14.103.1:34111)
I20260812 02:32:50.316972 26283 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.103.1@KRBTEST.COM'} at 127.14.103.1:52327
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.103.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.2.keytab.
Entry for principal kudu/127.14.103.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.2.keytab.
I20260812 02:32:50.336581 14748 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbep4M4/build/debug/bin/kudu
/tmp/dist-test-taskbep4M4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501898713699-14748-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501898713699-14748-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501898713699-14748-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501898713699-14748-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.103.2:0
--local_ip_for_outbound_sockets=127.14.103.2
--webserver_interface=127.14.103.2
--webserver_port=0
--tserver_master_addrs=127.14.103.62:42157
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.2.keytab
--principal=kudu/127.14.103.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.103.20:36113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501898713699-14748-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501898713699-14748-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501898713699-14748-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:32:50.458752 26513 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:32:50.459157 26513 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:32:50.459213 26513 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:32:50.463440 26513 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:32:50.463524 26513 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:32:50.463587 26513 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.103.2
I20260812 02:32:50.469305 26513 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.103.20:36113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501898713699-14748-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501898713699-14748-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.2.keytab
--principal=kudu/127.14.103.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.103.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501898713699-14748-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.103.2
--webserver_port=0
--tserver_master_addrs=127.14.103.62:42157
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26513
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.103.2
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501898713699-14748-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:32:50.470503 26513 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:32:50.471623 26513 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:32:50.482712 26518 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:32:50.482758 26521 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:32:50.482831 26513 server_base.cc:1034] running on GCE node
W20260812 02:32:50.482725 26519 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:32:50.483609 26513 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:32:50.484326 26513 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:32:50.485502 26513 hybrid_clock.cc:648] HybridClock initialized: now 1786501970485486 us; error 33 us; skew 500 ppm
Aug 12 02:32:50 dist-test-slave-2w3w krb5kdc[26232](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501970, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:32:50.489111 26513 init.cc:377] Logged in from keytab as kudu/127.14.103.2@KRBTEST.COM (short username kudu)
I20260812 02:32:50.490259 26513 webserver.cc:460] Webserver started at http://127.14.103.2:44081/ using document root <none> and password file <none>
I20260812 02:32:50.490931 26513 fs_manager.cc:359] Metadata directory not provided
I20260812 02:32:50.491081 26513 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:32:50.491410 26513 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:32:50.493706 26513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501898713699-14748-0/minicluster-data/ts-1/data/instance:
uuid: "d588d8fb3a7e42e7807b5dd57e2bf714"
format_stamp: "Formatted at 2026-08-12 02:32:50 on dist-test-slave-2w3w"
server_key: "680072f2391452354cbc6f79d3e8d105"
server_key_iv: "086ec8ce31ca04b0935e8005080f749c"
server_key_version: "encryptionkey@0"
I20260812 02:32:50.494294 26513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501898713699-14748-0/minicluster-data/ts-1/wal/instance:
uuid: "d588d8fb3a7e42e7807b5dd57e2bf714"
format_stamp: "Formatted at 2026-08-12 02:32:50 on dist-test-slave-2w3w"
server_key: "680072f2391452354cbc6f79d3e8d105"
server_key_iv: "086ec8ce31ca04b0935e8005080f749c"
server_key_version: "encryptionkey@0"
I20260812 02:32:50.498546 26513 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260812 02:32:50.501511 26528 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:32:50.503065 26513 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 02:32:50.503248 26513 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501898713699-14748-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501898713699-14748-0/minicluster-data/ts-1/wal
uuid: "d588d8fb3a7e42e7807b5dd57e2bf714"
format_stamp: "Formatted at 2026-08-12 02:32:50 on dist-test-slave-2w3w"
server_key: "680072f2391452354cbc6f79d3e8d105"
server_key_iv: "086ec8ce31ca04b0935e8005080f749c"
server_key_version: "encryptionkey@0"
I20260812 02:32:50.503361 26513 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501898713699-14748-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501898713699-14748-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501898713699-14748-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:32:50.532040 26513 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:32:50.535856 26513 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:32:50.536092 26513 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:32:50.536795 26513 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:32:50.537750 26513 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:32:50.537802 26513 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:32:50.537844 26513 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:32:50.537902 26513 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:32:50.547657 26513 rpc_server.cc:307] RPC server started. Bound to: 127.14.103.2:33065
I20260812 02:32:50.547752 26663 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.103.2:33065 every 8 connection(s)
I20260812 02:32:50.548671 26513 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501898713699-14748-0/minicluster-data/ts-1/data/info.pb
I20260812 02:32:50.556082 14748 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbep4M4/build/debug/bin/kudu as pid 26513
I20260812 02:32:50.556259 14748 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501898713699-14748-0/minicluster-data/ts-1/wal/instance
I20260812 02:32:50.556602 14748 external_mini_cluster.cc:1428] Setting key 422a58d8133e781f66964553f9c2fb2f
Aug 12 02:32:50 dist-test-slave-2w3w krb5kdc[26232](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501970, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.2@KRBTEST.COM for kudu/127.14.103.62@KRBTEST.COM
I20260812 02:32:50.565011 26326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:50.550739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.2:35901 (local address 127.14.103.62:42157)
0812 02:32:50.550892 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:50.550897 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:50.551793 (+ 896us) server_negotiation.cc:407] Connection header received
0812 02:32:50.553184 (+ 1391us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:50.553187 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:50.553236 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:50.553313 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:32:50.554568 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:50.555148 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:50.556406 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:50.556777 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:50.556894 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:50.559846 (+ 2952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:32:50.559853 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:32:50.559859 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:32:50.559896 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:32:50.561711 (+ 1815us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:50.562339 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:50.562347 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:50.562351 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:50.562474 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:50.562949 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:50.562955 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:50.562977 (+ 22us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:50.563295 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:32:50.563448 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0812 02:32:50.564548 (+ 1100us) server_negotiation.cc:299] Negotiation successful
0812 02:32:50.564745 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260812 02:32:50.566393 26666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:50.551132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:42157 (local address 127.14.103.2:35901)
0812 02:32:50.551627 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:50.551671 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0812 02:32:50.552981 (+ 1310us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:32:50.553482 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:32:50.553488 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:32:50.553913 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:32:50.554408 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:50.554422 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:50.555277 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:50.555281 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:50.556231 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:50.556278 (+ 47us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:50.556966 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:50.556969 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:50.557189 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:50.557955 (+ 766us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:32:50.557980 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:32:50.559646 (+ 1666us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:32:50.561870 (+ 2224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:32:50.561877 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:32:50.561893 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:32:50.562206 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:32:50.562621 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:32:50.562624 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:32:50.562625 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:32:50.562801 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:32:50.563476 (+ 675us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:32:50.563482 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:32:50.563879 (+ 397us) client_negotiation.cc:769] Sending connection context
0812 02:32:50.564381 (+ 502us) client_negotiation.cc:240] Negotiation successful
0812 02:32:50.564750 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":120,"threads_started":1}
I20260812 02:32:50.568142 26664 heartbeater.cc:347] Connected to a master server at 127.14.103.62:42157
I20260812 02:32:50.568475 26664 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.14.103.3@KRBTEST.COM; defaulting to no policy
I20260812 02:32:50.569175 26664 heartbeater.cc:511] Master 127.14.103.62:42157 requested a full tablet report, sending...
I20260812 02:32:50.570823 26283 ts_manager.cc:194] Registered new tserver with Master: d588d8fb3a7e42e7807b5dd57e2bf714 (127.14.103.2:33065)
I20260812 02:32:50.571590 26283 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.103.2@KRBTEST.COM'} at 127.14.103.2:35901
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.103.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.103.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.3.keytab.
Entry for principal kudu/127.14.103.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.3.keytab.
I20260812 02:32:50.595469 14748 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbep4M4/build/debug/bin/kudu
/tmp/dist-test-taskbep4M4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501898713699-14748-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501898713699-14748-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501898713699-14748-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501898713699-14748-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.103.3:0
--local_ip_for_outbound_sockets=127.14.103.3
--webserver_interface=127.14.103.3
--webserver_port=0
--tserver_master_addrs=127.14.103.62:42157
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.3.keytab
--principal=kudu/127.14.103.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.103.20:36113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501898713699-14748-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501898713699-14748-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501898713699-14748-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:32:50.716837 26670 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:32:50.717199 26670 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:32:50.717281 26670 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:32:50.721798 26670 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:32:50.721963 26670 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:32:50.722071 26670 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.103.3
I20260812 02:32:50.728085 26670 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.103.20:36113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501898713699-14748-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501898713699-14748-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.3.keytab
--principal=kudu/127.14.103.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.103.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501898713699-14748-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.103.3
--webserver_port=0
--tserver_master_addrs=127.14.103.62:42157
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26670
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.103.3
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501898713699-14748-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:32:50.729311 26670 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:32:50.730429 26670 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:32:50.741720 26676 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:32:50.741730 26678 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:32:50.741739 26675 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:32:50.741843 26670 server_base.cc:1034] running on GCE node
I20260812 02:32:50.742607 26670 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:32:50.743465 26670 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:32:50.744660 26670 hybrid_clock.cc:648] HybridClock initialized: now 1786501970744623 us; error 52 us; skew 500 ppm
Aug 12 02:32:50 dist-test-slave-2w3w krb5kdc[26232](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501970, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:32:50.748310 26670 init.cc:377] Logged in from keytab as kudu/127.14.103.3@KRBTEST.COM (short username kudu)
I20260812 02:32:50.749451 26670 webserver.cc:460] Webserver started at http://127.14.103.3:41109/ using document root <none> and password file <none>
I20260812 02:32:50.750077 26670 fs_manager.cc:359] Metadata directory not provided
I20260812 02:32:50.750149 26670 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:32:50.750403 26670 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:32:50.752792 26670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501898713699-14748-0/minicluster-data/ts-2/data/instance:
uuid: "84d94e10133a4442ba7973d269b33dc3"
format_stamp: "Formatted at 2026-08-12 02:32:50 on dist-test-slave-2w3w"
server_key: "7de56c871b7290604a6533e630f0e8e2"
server_key_iv: "19795898e09ed78514f8bf54e965d55b"
server_key_version: "encryptionkey@0"
I20260812 02:32:50.753424 26670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501898713699-14748-0/minicluster-data/ts-2/wal/instance:
uuid: "84d94e10133a4442ba7973d269b33dc3"
format_stamp: "Formatted at 2026-08-12 02:32:50 on dist-test-slave-2w3w"
server_key: "7de56c871b7290604a6533e630f0e8e2"
server_key_iv: "19795898e09ed78514f8bf54e965d55b"
server_key_version: "encryptionkey@0"
I20260812 02:32:50.757874 26670 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260812 02:32:50.761010 26685 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:32:50.762639 26670 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260812 02:32:50.762899 26670 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501898713699-14748-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501898713699-14748-0/minicluster-data/ts-2/wal
uuid: "84d94e10133a4442ba7973d269b33dc3"
format_stamp: "Formatted at 2026-08-12 02:32:50 on dist-test-slave-2w3w"
server_key: "7de56c871b7290604a6533e630f0e8e2"
server_key_iv: "19795898e09ed78514f8bf54e965d55b"
server_key_version: "encryptionkey@0"
I20260812 02:32:50.763126 26670 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501898713699-14748-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501898713699-14748-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501898713699-14748-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:32:50.798516 26670 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:32:50.802531 26670 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:32:50.802754 26670 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:32:50.803467 26670 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:32:50.804344 26670 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:32:50.804392 26670 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:32:50.804435 26670 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:32:50.804450 26670 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:32:50.813899 26670 rpc_server.cc:307] RPC server started. Bound to: 127.14.103.3:36655
I20260812 02:32:50.813927 26820 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.103.3:36655 every 8 connection(s)
I20260812 02:32:50.814920 26670 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501898713699-14748-0/minicluster-data/ts-2/data/info.pb
I20260812 02:32:50.815480 14748 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbep4M4/build/debug/bin/kudu as pid 26670
I20260812 02:32:50.815618 14748 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501898713699-14748-0/minicluster-data/ts-2/wal/instance
I20260812 02:32:50.816004 14748 external_mini_cluster.cc:1428] Setting key 57cf46ad3158ba4a604f19cc1adac2c8
Aug 12 02:32:50 dist-test-slave-2w3w krb5kdc[26232](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501970, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.3@KRBTEST.COM for kudu/127.14.103.62@KRBTEST.COM
I20260812 02:32:50.832362 26326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:50.817139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.3:49515 (local address 127.14.103.62:42157)
0812 02:32:50.817299 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:50.817303 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:50.818428 (+ 1125us) server_negotiation.cc:407] Connection header received
0812 02:32:50.820480 (+ 2052us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:50.820483 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:50.820541 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:50.820627 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:32:50.822083 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:50.822726 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:50.824298 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:50.824538 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:50.824637 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:50.827843 (+ 3206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:32:50.827846 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:32:50.827849 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:32:50.827880 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:32:50.829535 (+ 1655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:50.830305 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:50.830308 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:50.830309 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:50.830393 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:50.830896 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:50.830900 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:50.830901 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:50.831292 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:32:50.831438 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0812 02:32:50.832037 (+ 599us) server_negotiation.cc:299] Negotiation successful
0812 02:32:50.832186 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260812 02:32:50.833281 26824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:50.817668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:42157 (local address 127.14.103.3:49515)
0812 02:32:50.818252 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:50.818317 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0812 02:32:50.820190 (+ 1873us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:32:50.820913 (+ 723us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:32:50.820921 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:32:50.821358 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:32:50.821902 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:50.821917 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:50.822955 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:50.823025 (+ 70us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:50.824095 (+ 1070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:50.824107 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:50.824710 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:50.824713 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:50.824918 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:50.825572 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:32:50.825589 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:32:50.827588 (+ 1999us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:32:50.829693 (+ 2105us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:32:50.829699 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:32:50.829716 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:32:50.830159 (+ 443us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:32:50.830504 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:32:50.830507 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:32:50.830509 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:32:50.830723 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:32:50.831439 (+ 716us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:32:50.831447 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:32:50.831749 (+ 302us) client_negotiation.cc:769] Sending connection context
0812 02:32:50.831952 (+ 203us) client_negotiation.cc:240] Negotiation successful
0812 02:32:50.832331 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":130,"threads_started":1}
I20260812 02:32:50.834599 26821 heartbeater.cc:347] Connected to a master server at 127.14.103.62:42157
I20260812 02:32:50.834872 26821 heartbeater.cc:464] Registering TS with master...
I20260812 02:32:50.835474 26821 heartbeater.cc:511] Master 127.14.103.62:42157 requested a full tablet report, sending...
I20260812 02:32:50.836563 26283 ts_manager.cc:194] Registered new tserver with Master: 84d94e10133a4442ba7973d269b33dc3 (127.14.103.3:36655)
I20260812 02:32:50.837211 26283 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.103.3@KRBTEST.COM'} at 127.14.103.3:49515
I20260812 02:32:50.842590 14748 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 02:32:50.853153 26326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:50.844834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51314 (local address 127.14.103.62:42157)
0812 02:32:50.845016 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:50.845029 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:50.845204 (+ 175us) server_negotiation.cc:407] Connection header received
0812 02:32:50.845303 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:50.845305 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:50.845352 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:50.845426 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:32:50.846227 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:50.846760 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:50.847844 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:50.848087 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:50.848191 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:50.849785 (+ 1594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:32:50.849794 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:32:50.849797 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:32:50.849826 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:32:50.851419 (+ 1593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:50.851884 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:50.851887 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:50.851888 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:50.851965 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:32:50.852231 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:32:50.852233 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:32:50.852234 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:32:50.852461 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:32:50.852568 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0812 02:32:50.852813 (+ 245us) server_negotiation.cc:299] Negotiation successful
0812 02:32:50.852940 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260812 02:32:50.856426 26283 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51314:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 02:32:50.859582 26283 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 02:32:50.875841 26839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:50.870179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47110 (local address 127.14.103.1:34111)
0812 02:32:50.870465 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:50.870473 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:50.870727 (+ 254us) server_negotiation.cc:407] Connection header received
0812 02:32:50.871011 (+ 284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:50.871017 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:50.871195 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:50.871337 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:32:50.871805 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:50.872878 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:50.874403 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:50.875102 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:50.875228 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:50.875393 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0812 02:32:50.875559 (+ 166us) server_negotiation.cc:299] Negotiation successful
0812 02:32:50.875668 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":94,"threads_started":1}
I20260812 02:32:50.875855 26834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:50.869237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.2:33065 (local address 127.0.0.1:41478)
0812 02:32:50.870125 (+ 888us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:50.870140 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 02:32:50.870231 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:32:50.871435 (+ 1204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:32:50.871438 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:32:50.871469 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:32:50.871668 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:50.871676 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:50.872845 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:50.872849 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:50.874101 (+ 1252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:50.874110 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:50.875315 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:50.875319 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:50.875437 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:50.875468 (+ 31us) client_negotiation.cc:769] Sending connection context
0812 02:32:50.875520 (+ 52us) client_negotiation.cc:240] Negotiation successful
0812 02:32:50.875689 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":822,"mutex_wait_us":63,"thread_start_us":78,"threads_started":1}
I20260812 02:32:50.875830 26838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:50.869995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.1:34111 (local address 127.0.0.1:47110)
0812 02:32:50.870612 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:50.870624 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 02:32:50.870745 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:32:50.871479 (+ 734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:32:50.871481 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:32:50.871494 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:32:50.871671 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:50.871676 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:50.873007 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:50.873014 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:50.874237 (+ 1223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:50.874248 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:50.875228 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:50.875231 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:50.875341 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:50.875380 (+ 39us) client_negotiation.cc:769] Sending connection context
0812 02:32:50.875529 (+ 149us) client_negotiation.cc:240] Negotiation successful
0812 02:32:50.875681 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":557,"mutex_wait_us":124,"thread_start_us":88,"threads_started":1}
I20260812 02:32:50.876224 26837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:50.869929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49380 (local address 127.14.103.3:36655)
0812 02:32:50.870336 (+ 407us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:50.870350 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:50.870376 (+ 26us) server_negotiation.cc:407] Connection header received
0812 02:32:50.870491 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:50.870496 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:50.870729 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:50.870897 (+ 168us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:32:50.871407 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:50.872508 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:50.874378 (+ 1870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:50.875181 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:50.875288 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:50.875429 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0812 02:32:50.875933 (+ 504us) server_negotiation.cc:299] Negotiation successful
0812 02:32:50.876035 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":109,"threads_started":1}
I20260812 02:32:50.875855 26833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:50.869179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.3:36655 (local address 127.0.0.1:49380)
0812 02:32:50.869584 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:50.869643 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0812 02:32:50.869787 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:32:50.871124 (+ 1337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:32:50.871130 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:32:50.871152 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:32:50.871278 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:50.871284 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:50.872661 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:50.872664 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:50.874237 (+ 1573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:50.874248 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:50.875489 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:50.875495 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:50.875588 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:50.875609 (+ 21us) client_negotiation.cc:769] Sending connection context
0812 02:32:50.875677 (+ 68us) client_negotiation.cc:240] Negotiation successful
0812 02:32:50.875732 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":215,"mutex_wait_us":52,"thread_start_us":100,"threads_started":1}
I20260812 02:32:50.876431 26836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:50.869674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41478 (local address 127.14.103.2:33065)
0812 02:32:50.871028 (+ 1354us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:50.871039 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:50.871060 (+ 21us) server_negotiation.cc:407] Connection header received
0812 02:32:50.871125 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:50.871129 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:50.871287 (+ 158us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:50.871808 (+ 521us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:32:50.871944 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:50.872704 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:50.874229 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:50.875167 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:50.875931 (+ 764us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:50.876079 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0812 02:32:50.876130 (+ 51us) server_negotiation.cc:299] Negotiation successful
0812 02:32:50.876260 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1227,"thread_start_us":144,"threads_started":1}
I20260812 02:32:50.879598 26584 tablet_service.cc:1467] Processing CreateTablet for tablet 195703bec22d4ef4b075a0921d765ac2 (DEFAULT_TABLE table=test-table [id=212358d10d7c40478279dc7f109424b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:32:50.879598 26744 tablet_service.cc:1467] Processing CreateTablet for tablet 195703bec22d4ef4b075a0921d765ac2 (DEFAULT_TABLE table=test-table [id=212358d10d7c40478279dc7f109424b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:32:50.880769 26584 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 195703bec22d4ef4b075a0921d765ac2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:32:50.880775 26744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 195703bec22d4ef4b075a0921d765ac2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:32:50.881644 26431 tablet_service.cc:1467] Processing CreateTablet for tablet 195703bec22d4ef4b075a0921d765ac2 (DEFAULT_TABLE table=test-table [id=212358d10d7c40478279dc7f109424b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:32:50.882499 26431 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 195703bec22d4ef4b075a0921d765ac2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:32:50.888347 26840 tablet_bootstrap.cc:492] T 195703bec22d4ef4b075a0921d765ac2 P 19573b79ad2e48cba2e8fb0885ce88f9: Bootstrap starting.
I20260812 02:32:50.889076 26842 tablet_bootstrap.cc:492] T 195703bec22d4ef4b075a0921d765ac2 P 84d94e10133a4442ba7973d269b33dc3: Bootstrap starting.
I20260812 02:32:50.889212 26841 tablet_bootstrap.cc:492] T 195703bec22d4ef4b075a0921d765ac2 P d588d8fb3a7e42e7807b5dd57e2bf714: Bootstrap starting.
I20260812 02:32:50.891382 26842 tablet_bootstrap.cc:654] T 195703bec22d4ef4b075a0921d765ac2 P 84d94e10133a4442ba7973d269b33dc3: Neither blocks nor log segments found. Creating new log.
I20260812 02:32:50.891382 26840 tablet_bootstrap.cc:654] T 195703bec22d4ef4b075a0921d765ac2 P 19573b79ad2e48cba2e8fb0885ce88f9: Neither blocks nor log segments found. Creating new log.
I20260812 02:32:50.891855 26841 tablet_bootstrap.cc:654] T 195703bec22d4ef4b075a0921d765ac2 P d588d8fb3a7e42e7807b5dd57e2bf714: Neither blocks nor log segments found. Creating new log.
I20260812 02:32:50.892381 26840 log.cc:824] T 195703bec22d4ef4b075a0921d765ac2 P 19573b79ad2e48cba2e8fb0885ce88f9: Log is configured to *not* fsync() on all Append() calls
I20260812 02:32:50.892697 26841 log.cc:824] T 195703bec22d4ef4b075a0921d765ac2 P d588d8fb3a7e42e7807b5dd57e2bf714: Log is configured to *not* fsync() on all Append() calls
I20260812 02:32:50.893172 26842 log.cc:824] T 195703bec22d4ef4b075a0921d765ac2 P 84d94e10133a4442ba7973d269b33dc3: Log is configured to *not* fsync() on all Append() calls
I20260812 02:32:50.894634 26841 tablet_bootstrap.cc:492] T 195703bec22d4ef4b075a0921d765ac2 P d588d8fb3a7e42e7807b5dd57e2bf714: No bootstrap required, opened a new log
I20260812 02:32:50.894716 26840 tablet_bootstrap.cc:492] T 195703bec22d4ef4b075a0921d765ac2 P 19573b79ad2e48cba2e8fb0885ce88f9: No bootstrap required, opened a new log
I20260812 02:32:50.894887 26841 ts_tablet_manager.cc:1397] T 195703bec22d4ef4b075a0921d765ac2 P d588d8fb3a7e42e7807b5dd57e2bf714: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260812 02:32:50.894938 26840 ts_tablet_manager.cc:1397] T 195703bec22d4ef4b075a0921d765ac2 P 19573b79ad2e48cba2e8fb0885ce88f9: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260812 02:32:50.895138 26842 tablet_bootstrap.cc:492] T 195703bec22d4ef4b075a0921d765ac2 P 84d94e10133a4442ba7973d269b33dc3: No bootstrap required, opened a new log
I20260812 02:32:50.895299 26842 ts_tablet_manager.cc:1397] T 195703bec22d4ef4b075a0921d765ac2 P 84d94e10133a4442ba7973d269b33dc3: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260812 02:32:50.897892 26841 raft_consensus.cc:348] T 195703bec22d4ef4b075a0921d765ac2 P d588d8fb3a7e42e7807b5dd57e2bf714 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84d94e10133a4442ba7973d269b33dc3" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 36655 } } peers { permanent_uuid: "d588d8fb3a7e42e7807b5dd57e2bf714" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 33065 } } peers { permanent_uuid: "19573b79ad2e48cba2e8fb0885ce88f9" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 34111 } }
I20260812 02:32:50.898125 26841 raft_consensus.cc:374] T 195703bec22d4ef4b075a0921d765ac2 P d588d8fb3a7e42e7807b5dd57e2bf714 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:32:50.898157 26841 raft_consensus.cc:729] T 195703bec22d4ef4b075a0921d765ac2 P d588d8fb3a7e42e7807b5dd57e2bf714 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d588d8fb3a7e42e7807b5dd57e2bf714, State: Initialized, Role: FOLLOWER
I20260812 02:32:50.898619 26841 consensus_queue.cc:260] T 195703bec22d4ef4b075a0921d765ac2 P d588d8fb3a7e42e7807b5dd57e2bf714 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84d94e10133a4442ba7973d269b33dc3" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 36655 } } peers { permanent_uuid: "d588d8fb3a7e42e7807b5dd57e2bf714" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 33065 } } peers { permanent_uuid: "19573b79ad2e48cba2e8fb0885ce88f9" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 34111 } }
I20260812 02:32:50.899293 26842 raft_consensus.cc:348] T 195703bec22d4ef4b075a0921d765ac2 P 84d94e10133a4442ba7973d269b33dc3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84d94e10133a4442ba7973d269b33dc3" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 36655 } } peers { permanent_uuid: "d588d8fb3a7e42e7807b5dd57e2bf714" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 33065 } } peers { permanent_uuid: "19573b79ad2e48cba2e8fb0885ce88f9" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 34111 } }
I20260812 02:32:50.899559 26842 raft_consensus.cc:374] T 195703bec22d4ef4b075a0921d765ac2 P 84d94e10133a4442ba7973d269b33dc3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:32:50.899420 26840 raft_consensus.cc:348] T 195703bec22d4ef4b075a0921d765ac2 P 19573b79ad2e48cba2e8fb0885ce88f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84d94e10133a4442ba7973d269b33dc3" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 36655 } } peers { permanent_uuid: "d588d8fb3a7e42e7807b5dd57e2bf714" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 33065 } } peers { permanent_uuid: "19573b79ad2e48cba2e8fb0885ce88f9" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 34111 } }
I20260812 02:32:50.899606 26842 raft_consensus.cc:729] T 195703bec22d4ef4b075a0921d765ac2 P 84d94e10133a4442ba7973d269b33dc3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84d94e10133a4442ba7973d269b33dc3, State: Initialized, Role: FOLLOWER
I20260812 02:32:50.899632 26840 raft_consensus.cc:374] T 195703bec22d4ef4b075a0921d765ac2 P 19573b79ad2e48cba2e8fb0885ce88f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:32:50.899688 26840 raft_consensus.cc:729] T 195703bec22d4ef4b075a0921d765ac2 P 19573b79ad2e48cba2e8fb0885ce88f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19573b79ad2e48cba2e8fb0885ce88f9, State: Initialized, Role: FOLLOWER
I20260812 02:32:50.899770 26841 ts_tablet_manager.cc:1428] T 195703bec22d4ef4b075a0921d765ac2 P d588d8fb3a7e42e7807b5dd57e2bf714: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260812 02:32:50.900043 26664 heartbeater.cc:503] Master 127.14.103.62:42157 was elected leader, sending a full tablet report...
I20260812 02:32:50.900219 26840 consensus_queue.cc:260] T 195703bec22d4ef4b075a0921d765ac2 P 19573b79ad2e48cba2e8fb0885ce88f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84d94e10133a4442ba7973d269b33dc3" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 36655 } } peers { permanent_uuid: "d588d8fb3a7e42e7807b5dd57e2bf714" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 33065 } } peers { permanent_uuid: "19573b79ad2e48cba2e8fb0885ce88f9" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 34111 } }
I20260812 02:32:50.900688 26842 consensus_queue.cc:260] T 195703bec22d4ef4b075a0921d765ac2 P 84d94e10133a4442ba7973d269b33dc3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84d94e10133a4442ba7973d269b33dc3" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 36655 } } peers { permanent_uuid: "d588d8fb3a7e42e7807b5dd57e2bf714" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 33065 } } peers { permanent_uuid: "19573b79ad2e48cba2e8fb0885ce88f9" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 34111 } }
I20260812 02:32:50.900964 26507 heartbeater.cc:503] Master 127.14.103.62:42157 was elected leader, sending a full tablet report...
I20260812 02:32:50.901324 26840 ts_tablet_manager.cc:1428] T 195703bec22d4ef4b075a0921d765ac2 P 19573b79ad2e48cba2e8fb0885ce88f9: Time spent starting tablet: real 0.006s user 0.004s sys 0.002s
I20260812 02:32:50.901693 26842 ts_tablet_manager.cc:1428] T 195703bec22d4ef4b075a0921d765ac2 P 84d94e10133a4442ba7973d269b33dc3: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260812 02:32:50.901785 26821 heartbeater.cc:503] Master 127.14.103.62:42157 was elected leader, sending a full tablet report...
I20260812 02:32:50.934425 26848 raft_consensus.cc:482] T 195703bec22d4ef4b075a0921d765ac2 P 19573b79ad2e48cba2e8fb0885ce88f9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 02:32:50.934684 26848 raft_consensus.cc:504] T 195703bec22d4ef4b075a0921d765ac2 P 19573b79ad2e48cba2e8fb0885ce88f9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84d94e10133a4442ba7973d269b33dc3" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 36655 } } peers { permanent_uuid: "d588d8fb3a7e42e7807b5dd57e2bf714" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 33065 } } peers { permanent_uuid: "19573b79ad2e48cba2e8fb0885ce88f9" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 34111 } }
I20260812 02:32:50.935979 26848 leader_election.cc:288] T 195703bec22d4ef4b075a0921d765ac2 P 19573b79ad2e48cba2e8fb0885ce88f9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 84d94e10133a4442ba7973d269b33dc3 (127.14.103.3:36655), d588d8fb3a7e42e7807b5dd57e2bf714 (127.14.103.2:33065)
I20260812 02:32:50.940448 26837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:50.936600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.1:38807 (local address 127.14.103.3:36655)
0812 02:32:50.936810 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:50.936815 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:50.936973 (+ 158us) server_negotiation.cc:407] Connection header received
0812 02:32:50.937133 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:50.937136 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:50.937195 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:50.937289 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:32:50.937643 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:50.938177 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:50.939425 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:50.939842 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:50.939979 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:50.940036 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0812 02:32:50.940213 (+ 177us) server_negotiation.cc:299] Negotiation successful
0812 02:32:50.940285 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
I20260812 02:32:50.940483 26836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:50.936583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.1:47549 (local address 127.14.103.2:33065)
0812 02:32:50.936743 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:50.936748 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:50.937013 (+ 265us) server_negotiation.cc:407] Connection header received
0812 02:32:50.937285 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:50.937287 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:50.937375 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:50.937500 (+ 125us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:32:50.937784 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:50.938333 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:50.939492 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:50.939902 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:50.940039 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:50.940099 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0812 02:32:50.940270 (+ 171us) server_negotiation.cc:299] Negotiation successful
0812 02:32:50.940349 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260812 02:32:50.940515 26852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:50.936333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.3:36655 (local address 127.14.103.1:38807)
0812 02:32:50.936846 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:50.936869 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0812 02:32:50.937012 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:32:50.937354 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:32:50.937357 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:32:50.937377 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:32:50.937520 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:50.937526 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:50.938326 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:50.938329 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:50.939300 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:50.939309 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:50.939975 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:50.939979 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:50.940079 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:50.940108 (+ 29us) client_negotiation.cc:769] Sending connection context
0812 02:32:50.940198 (+ 90us) client_negotiation.cc:240] Negotiation successful
0812 02:32:50.940278 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":379,"mutex_wait_us":1,"thread_start_us":159,"threads_started":1}
I20260812 02:32:50.940528 26850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:50.936333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.2:33065 (local address 127.14.103.1:47549)
0812 02:32:50.936925 (+ 592us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:50.936939 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 02:32:50.937075 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:32:50.937513 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:32:50.937515 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:32:50.937534 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:32:50.937660 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:50.937666 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:50.938460 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:50.938464 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:50.939355 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:32:50.939365 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:32:50.940029 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:32:50.940033 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:32:50.940140 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:50.940164 (+ 24us) client_negotiation.cc:769] Sending connection context
0812 02:32:50.940254 (+ 90us) client_negotiation.cc:240] Negotiation successful
0812 02:32:50.940330 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"mutex_wait_us":79,"thread_start_us":132,"threads_started":1}
I20260812 02:32:50.941394 26768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "195703bec22d4ef4b075a0921d765ac2" candidate_uuid: "19573b79ad2e48cba2e8fb0885ce88f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84d94e10133a4442ba7973d269b33dc3" is_pre_election: true
I20260812 02:32:50.941422 26608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "195703bec22d4ef4b075a0921d765ac2" candidate_uuid: "19573b79ad2e48cba2e8fb0885ce88f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d588d8fb3a7e42e7807b5dd57e2bf714" is_pre_election: true
I20260812 02:32:50.941771 26608 raft_consensus.cc:2454] T 195703bec22d4ef4b075a0921d765ac2 P d588d8fb3a7e42e7807b5dd57e2bf714 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 19573b79ad2e48cba2e8fb0885ce88f9 in term 0.
I20260812 02:32:50.941771 26768 raft_consensus.cc:2454] T 195703bec22d4ef4b075a0921d765ac2 P 84d94e10133a4442ba7973d269b33dc3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 19573b79ad2e48cba2e8fb0885ce88f9 in term 0.
I20260812 02:32:50.942360 26376 leader_election.cc:302] T 195703bec22d4ef4b075a0921d765ac2 P 19573b79ad2e48cba2e8fb0885ce88f9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 19573b79ad2e48cba2e8fb0885ce88f9, 84d94e10133a4442ba7973d269b33dc3, d588d8fb3a7e42e7807b5dd57e2bf714; no voters:
I20260812 02:32:50.942641 26848 raft_consensus.cc:2781] T 195703bec22d4ef4b075a0921d765ac2 P 19573b79ad2e48cba2e8fb0885ce88f9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 02:32:50.942726 26848 raft_consensus.cc:482] T 195703bec22d4ef4b075a0921d765ac2 P 19573b79ad2e48cba2e8fb0885ce88f9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 02:32:50.942770 26848 raft_consensus.cc:3037] T 195703bec22d4ef4b075a0921d765ac2 P 19573b79ad2e48cba2e8fb0885ce88f9 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:32:50.943953 26848 raft_consensus.cc:504] T 195703bec22d4ef4b075a0921d765ac2 P 19573b79ad2e48cba2e8fb0885ce88f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84d94e10133a4442ba7973d269b33dc3" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 36655 } } peers { permanent_uuid: "d588d8fb3a7e42e7807b5dd57e2bf714" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 33065 } } peers { permanent_uuid: "19573b79ad2e48cba2e8fb0885ce88f9" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 34111 } }
I20260812 02:32:50.944480 26848 leader_election.cc:288] T 195703bec22d4ef4b075a0921d765ac2 P 19573b79ad2e48cba2e8fb0885ce88f9 [CANDIDATE]: Term 1 election: Requested vote from peers 84d94e10133a4442ba7973d269b33dc3 (127.14.103.3:36655), d588d8fb3a7e42e7807b5dd57e2bf714 (127.14.103.2:33065)
I20260812 02:32:50.944814 26768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "195703bec22d4ef4b075a0921d765ac2" candidate_uuid: "19573b79ad2e48cba2e8fb0885ce88f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84d94e10133a4442ba7973d269b33dc3"
I20260812 02:32:50.944969 26608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "195703bec22d4ef4b075a0921d765ac2" candidate_uuid: "19573b79ad2e48cba2e8fb0885ce88f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d588d8fb3a7e42e7807b5dd57e2bf714"
I20260812 02:32:50.944983 26768 raft_consensus.cc:3037] T 195703bec22d4ef4b075a0921d765ac2 P 84d94e10133a4442ba7973d269b33dc3 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:32:50.945169 26608 raft_consensus.cc:3037] T 195703bec22d4ef4b075a0921d765ac2 P d588d8fb3a7e42e7807b5dd57e2bf714 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:32:50.946166 26768 raft_consensus.cc:2454] T 195703bec22d4ef4b075a0921d765ac2 P 84d94e10133a4442ba7973d269b33dc3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19573b79ad2e48cba2e8fb0885ce88f9 in term 1.
I20260812 02:32:50.946460 26608 raft_consensus.cc:2454] T 195703bec22d4ef4b075a0921d765ac2 P d588d8fb3a7e42e7807b5dd57e2bf714 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19573b79ad2e48cba2e8fb0885ce88f9 in term 1.
I20260812 02:32:50.946538 26379 leader_election.cc:302] T 195703bec22d4ef4b075a0921d765ac2 P 19573b79ad2e48cba2e8fb0885ce88f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19573b79ad2e48cba2e8fb0885ce88f9, 84d94e10133a4442ba7973d269b33dc3; no voters:
I20260812 02:32:50.946900 26848 raft_consensus.cc:2781] T 195703bec22d4ef4b075a0921d765ac2 P 19573b79ad2e48cba2e8fb0885ce88f9 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:32:50.947194 26848 raft_consensus.cc:686] T 195703bec22d4ef4b075a0921d765ac2 P 19573b79ad2e48cba2e8fb0885ce88f9 [term 1 LEADER]: Becoming Leader. State: Replica: 19573b79ad2e48cba2e8fb0885ce88f9, State: Running, Role: LEADER
I20260812 02:32:50.947616 26848 consensus_queue.cc:237] T 195703bec22d4ef4b075a0921d765ac2 P 19573b79ad2e48cba2e8fb0885ce88f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84d94e10133a4442ba7973d269b33dc3" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 36655 } } peers { permanent_uuid: "d588d8fb3a7e42e7807b5dd57e2bf714" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 33065 } } peers { permanent_uuid: "19573b79ad2e48cba2e8fb0885ce88f9" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 34111 } }
I20260812 02:32:50.951095 26281 catalog_manager.cc:5495] T 195703bec22d4ef4b075a0921d765ac2 P 19573b79ad2e48cba2e8fb0885ce88f9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 19573b79ad2e48cba2e8fb0885ce88f9 (127.14.103.1). New cstate: current_term: 1 leader_uuid: "19573b79ad2e48cba2e8fb0885ce88f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84d94e10133a4442ba7973d269b33dc3" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 36655 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d588d8fb3a7e42e7807b5dd57e2bf714" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 33065 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "19573b79ad2e48cba2e8fb0885ce88f9" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 34111 } health_report { overall_health: HEALTHY } } }
I20260812 02:32:50.977777 26839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:50.973377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47120 (local address 127.14.103.1:34111)
0812 02:32:50.973528 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0812 02:32:50.973532 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:32:50.973564 (+ 32us) server_negotiation.cc:407] Connection header received
0812 02:32:50.973685 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:32:50.973691 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:32:50.973774 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:32:50.973889 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 02:32:50.974238 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:50.975056 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:50.976054 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:32:50.976391 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:32:50.976509 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:32:50.976865 (+ 356us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 02:32:50.977371 (+ 506us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 02:32:50.977491 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0812 02:32:50.977557 (+ 66us) server_negotiation.cc:299] Negotiation successful
0812 02:32:50.977626 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260812 02:32:50.984210 26608 raft_consensus.cc:1261] T 195703bec22d4ef4b075a0921d765ac2 P d588d8fb3a7e42e7807b5dd57e2bf714 [term 1 FOLLOWER]: Refusing update from remote peer 19573b79ad2e48cba2e8fb0885ce88f9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:32:50.984211 26768 raft_consensus.cc:1261] T 195703bec22d4ef4b075a0921d765ac2 P 84d94e10133a4442ba7973d269b33dc3 [term 1 FOLLOWER]: Refusing update from remote peer 19573b79ad2e48cba2e8fb0885ce88f9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:32:50.985105 26853 consensus_queue.cc:1035] T 195703bec22d4ef4b075a0921d765ac2 P 19573b79ad2e48cba2e8fb0885ce88f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d588d8fb3a7e42e7807b5dd57e2bf714" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 33065 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:32:50.985370 26848 consensus_queue.cc:1035] T 195703bec22d4ef4b075a0921d765ac2 P 19573b79ad2e48cba2e8fb0885ce88f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84d94e10133a4442ba7973d269b33dc3" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 36655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:32:50.994139 26856 mvcc.cc:205] Tried to move back new op lower bound from 7317512073142706176 to 7317512073004625920. Current Snapshot: MvccSnapshot[applied={T|T < 7317512073142706176}]
I20260812 02:32:50.994462 26857 mvcc.cc:205] Tried to move back new op lower bound from 7317512073142706176 to 7317512073004625920. Current Snapshot: MvccSnapshot[applied={T|T < 7317512073142706176}]
I20260812 02:32:50.995280 26858 mvcc.cc:205] Tried to move back new op lower bound from 7317512073142706176 to 7317512073004625920. Current Snapshot: MvccSnapshot[applied={T|T < 7317512073142706176}]
I20260812 02:32:51.007855 26283 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51314:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 02:32:51.008087 26283 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51314:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 02:32:51.012084 26283 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9dbd1a3ca0254e8696c3ab074724f460: Sending DeleteTablet for 3 replicas of tablet 195703bec22d4ef4b075a0921d765ac2
I20260812 02:32:51.013239 26584 tablet_service.cc:1514] Processing DeleteTablet for tablet 195703bec22d4ef4b075a0921d765ac2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:32:51 UTC) from {username='kudu', principal='kudu/127.14.103.62@KRBTEST.COM'} at 127.0.0.1:41478
I20260812 02:32:51.013252 26744 tablet_service.cc:1514] Processing DeleteTablet for tablet 195703bec22d4ef4b075a0921d765ac2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:32:51 UTC) from {username='kudu', principal='kudu/127.14.103.62@KRBTEST.COM'} at 127.0.0.1:49380
I20260812 02:32:51.013827 26868 tablet_replica.cc:330] T 195703bec22d4ef4b075a0921d765ac2 P d588d8fb3a7e42e7807b5dd57e2bf714: stopping tablet replica
I20260812 02:32:51.013890 26869 tablet_replica.cc:330] T 195703bec22d4ef4b075a0921d765ac2 P 84d94e10133a4442ba7973d269b33dc3: stopping tablet replica
I20260812 02:32:51.013859 26431 tablet_service.cc:1514] Processing DeleteTablet for tablet 195703bec22d4ef4b075a0921d765ac2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:32:51 UTC) from {username='kudu', principal='kudu/127.14.103.62@KRBTEST.COM'} at 127.0.0.1:47110
I20260812 02:32:51.014305 26868 raft_consensus.cc:2229] T 195703bec22d4ef4b075a0921d765ac2 P d588d8fb3a7e42e7807b5dd57e2bf714 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 02:32:51.014308 26869 raft_consensus.cc:2229] T 195703bec22d4ef4b075a0921d765ac2 P 84d94e10133a4442ba7973d269b33dc3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 02:32:51.015008 26868 raft_consensus.cc:2258] T 195703bec22d4ef4b075a0921d765ac2 P d588d8fb3a7e42e7807b5dd57e2bf714 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:32:51.015373 14748 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbep4M4/build/debug/bin/kudu with pid 26358
I20260812 02:32:51.017900 26868 ts_tablet_manager.cc:1905] T 195703bec22d4ef4b075a0921d765ac2 P d588d8fb3a7e42e7807b5dd57e2bf714: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:32:51.019516 26869 raft_consensus.cc:2258] T 195703bec22d4ef4b075a0921d765ac2 P 84d94e10133a4442ba7973d269b33dc3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:32:51.021329 26868 ts_tablet_manager.cc:1918] T 195703bec22d4ef4b075a0921d765ac2 P d588d8fb3a7e42e7807b5dd57e2bf714: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 02:32:51.021477 26868 log.cc:1196] T 195703bec22d4ef4b075a0921d765ac2 P d588d8fb3a7e42e7807b5dd57e2bf714: Deleting WAL directory at /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501898713699-14748-0/minicluster-data/ts-1/wal/wals/195703bec22d4ef4b075a0921d765ac2
I20260812 02:32:51.021821 26868 ts_tablet_manager.cc:1939] T 195703bec22d4ef4b075a0921d765ac2 P d588d8fb3a7e42e7807b5dd57e2bf714: Deleting consensus metadata
I20260812 02:32:51.021910 26869 ts_tablet_manager.cc:1905] T 195703bec22d4ef4b075a0921d765ac2 P 84d94e10133a4442ba7973d269b33dc3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:32:51.023125 26265 catalog_manager.cc:4841] TS d588d8fb3a7e42e7807b5dd57e2bf714 (127.14.103.2:33065): tablet 195703bec22d4ef4b075a0921d765ac2 (table test-table [id=212358d10d7c40478279dc7f109424b7]) successfully deleted
I20260812 02:32:51.025512 26869 ts_tablet_manager.cc:1918] T 195703bec22d4ef4b075a0921d765ac2 P 84d94e10133a4442ba7973d269b33dc3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 02:32:51.025633 26869 log.cc:1196] T 195703bec22d4ef4b075a0921d765ac2 P 84d94e10133a4442ba7973d269b33dc3: Deleting WAL directory at /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786501898713699-14748-0/minicluster-data/ts-2/wal/wals/195703bec22d4ef4b075a0921d765ac2
I20260812 02:32:51.026011 26869 ts_tablet_manager.cc:1939] T 195703bec22d4ef4b075a0921d765ac2 P 84d94e10133a4442ba7973d269b33dc3: Deleting consensus metadata
I20260812 02:32:51.027456 26268 catalog_manager.cc:4841] TS 84d94e10133a4442ba7973d269b33dc3 (127.14.103.3:36655): tablet 195703bec22d4ef4b075a0921d765ac2 (table test-table [id=212358d10d7c40478279dc7f109424b7]) successfully deleted
W20260812 02:32:51.027477 26265 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.103.1:34111 (error 108)
I20260812 02:32:51.027621 14748 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbep4M4/build/debug/bin/kudu with pid 26513
I20260812 02:32:51.031505 26833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:32:51.031122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.1:34111 (local address 127.0.0.1:47124)
0812 02:32:51.031292 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0812 02:32:51.031385 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.1:34111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260812 02:32:51.031718 26265 catalog_manager.cc:4584] TS 19573b79ad2e48cba2e8fb0885ce88f9 (127.14.103.1:34111): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 195703bec22d4ef4b075a0921d765ac2: Network error: Client connection negotiation failed: client connection to 127.14.103.1:34111: connect: Connection refused (error 111)
I20260812 02:32:51.036043 14748 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbep4M4/build/debug/bin/kudu with pid 26670
I20260812 02:32:51.043345 14748 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbep4M4/build/debug/bin/kudu with pid 26247
2026-08-12T02:32:51Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2586 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501898713699-14748-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 02:32:51 dist-test-slave-2w3w krb5kdc[26875](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 02:32:51 dist-test-slave-2w3w krb5kdc[26875](info): set up 2 sockets
Aug 12 02:32:51 dist-test-slave-2w3w krb5kdc[26875](info): commencing operation
krb5kdc: starting...
W20260812 02:32:52.229815 14748 mini_kdc.cc:121] Time spent starting KDC: real 1.160s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501898713699-14748-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501898713699-14748-0/krb5kdc/test-user.keytab.
Aug 12 02:32:52 dist-test-slave-2w3w krb5kdc[26875](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786501972, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T02:32:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T02:32:52Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.14.103.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.14.103.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.14.103.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501898713699-14748-0/krb5kdc/rangeradmin_127.14.103.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.14.103.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501898713699-14748-0/krb5kdc/rangeradmin_127.14.103.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.14.103.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.14.103.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.14.103.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501898713699-14748-0/krb5kdc/rangerlookup_127.14.103.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.14.103.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501898713699-14748-0/krb5kdc/rangerlookup_127.14.103.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.14.103.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.14.103.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.103.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501898713699-14748-0/krb5kdc/HTTP_127.14.103.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.14.103.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501898713699-14748-0/krb5kdc/HTTP_127.14.103.20@KRBTEST.COM.keytab.
I20260812 02:32:52.460060 14748 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501898713699-14748-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskbep4M4/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501898713699-14748-0/minicluster-data/postgres -l logfile start
2026-08-12 02:32:55.900 UTC [26911] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-12 02:32:55.900 UTC [26911] LOG: listening on IPv4 address "127.14.103.20", port 43159
2026-08-12 02:32:55.907 UTC [26911] LOG: listening on Unix socket "/tmp/.s.PGSQL.43159"
2026-08-12 02:32:55.950 UTC [26917] LOG: database system was shut down at 2026-08-12 02:32:53 UTC
2026-08-12 02:32:55.956 UTC [26911] LOG: database system is ready to accept connections
I20260812 02:32:57.003634 14748 mini_postgres.cc:96] Postgres bound to 43159
2026-08-12 02:32:57.013 UTC [26927] FATAL: database "slave" does not exist
127.14.103.20:43159 - accepting connections
I20260812 02:32:57.014549 14748 mini_ranger.cc:161] Starting Ranger...
I20260812 02:32:57.033519 14748 mini_ranger.cc:84] Created miniranger Postgres user
I20260812 02:32:57.774389 14748 mini_ranger.cc:87] Created ranger Postgres database
I20260812 02:32:57.774513 14748 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501898713699-14748-0/minicluster-data/ranger-admin
2026-08-12 02:32:58,129 [I] DB FLAVOR :POSTGRES
2026-08-12 02:32:58,130 [I] --------- Verifying Ranger DB connection ---------
2026-08-12 02:32:58,130 [I] Checking connection..
2026-08-12 02:32:58,130 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbep4M4/build/debug/bin/postgresql.jar:/tmp/dist-test-taskbep4M4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.103.20:43159/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-12 02:32:58,426 [I] Checking connection passed.
2026-08-12 02:32:58,427 [I] --------- Verifying version history table ---------
2026-08-12 02:32:58,427 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbep4M4/build/debug/bin/postgresql.jar:/tmp/dist-test-taskbep4M4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.103.20:43159/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-12 02:32:58,731 [I] Table x_db_version_h does not exist in database ranger
2026-08-12 02:32:58,732 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbep4M4/build/debug/bin/postgresql.jar:/tmp/dist-test-taskbep4M4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.103.20:43159/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-12 02:32:59,004 [I] Table x_db_version_h does not exist in database ranger
2026-08-12 02:32:59,004 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-12 02:32:59,004 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbep4M4/build/debug/bin/postgresql.jar:/tmp/dist-test-taskbep4M4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.103.20:43159/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskbep4M4/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-12 02:32:59.286 UTC [27003] WARNING: there is no transaction in progress
2026-08-12 02:32:59,295 [I] create_dbversion_catalog.sql file imported successfully
2026-08-12 02:32:59,295 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbep4M4/build/debug/bin/postgresql.jar:/tmp/dist-test-taskbep4M4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.103.20:43159/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-12 02:32:59,583 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-12 02:32:59,584 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-12 02:32:59,584 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbep4M4/build/debug/bin/postgresql.jar:/tmp/dist-test-taskbep4M4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.103.20:43159/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-12 02:32:59,834 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbep4M4/build/debug/bin/postgresql.jar:/tmp/dist-test-taskbep4M4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.103.20:43159/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-12 02:33:00,125 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbep4M4/build/debug/bin/postgresql.jar:/tmp/dist-test-taskbep4M4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.103.20:43159/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-2w3w.c.gcp-upstream.internal','N') ;"
2026-08-12 02:33:00,385 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbep4M4/build/debug/bin/postgresql.jar:/tmp/dist-test-taskbep4M4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.103.20:43159/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-12 02:33:00,664 [I] Table x_portal_user does not exist in database ranger
2026-08-12 02:33:00,665 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbep4M4/build/debug/bin/postgresql.jar:/tmp/dist-test-taskbep4M4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.103.20:43159/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-12 02:33:00,953 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-12 02:33:00,953 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbep4M4/build/debug/bin/postgresql.jar:/tmp/dist-test-taskbep4M4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.103.20:43159/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-12 02:33:01,224 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbep4M4/build/debug/bin/postgresql.jar:/tmp/dist-test-taskbep4M4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.103.20:43159/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-12 02:33:01,521 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-12 02:33:01,521 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbep4M4/build/debug/bin/postgresql.jar:/tmp/dist-test-taskbep4M4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.103.20:43159/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskbep4M4/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-12 02:33:02.835 UTC [27132] WARNING: there is no transaction in progress
2026-08-12 02:33:02.857 UTC [27132] WARNING: there is no transaction in progress
2026-08-12 02:33:02.878 UTC [27132] WARNING: there is no transaction in progress
2026-08-12 02:33:02.900 UTC [27132] WARNING: there is no transaction in progress
2026-08-12 02:33:02.923 UTC [27132] WARNING: there is no transaction in progress
2026-08-12 02:33:02.946 UTC [27132] WARNING: there is no transaction in progress
2026-08-12 02:33:03.025 UTC [27132] WARNING: there is no transaction in progress
2026-08-12 02:33:03.034 UTC [27132] WARNING: there is no transaction in progress
2026-08-12 02:33:03.055 UTC [27132] WARNING: there is no transaction in progress
2026-08-12 02:33:03.067 UTC [27132] WARNING: there is no transaction in progress
2026-08-12 02:33:03.080 UTC [27132] WARNING: there is no transaction in progress
2026-08-12 02:33:03.100 UTC [27132] WARNING: there is no transaction in progress
2026-08-12 02:33:03.113 UTC [27132] WARNING: there is no transaction in progress
2026-08-12 02:33:03.121 UTC [27132] WARNING: there is no transaction in progress
2026-08-12 02:33:03.131 UTC [27132] WARNING: there is no transaction in progress
2026-08-12 02:33:03.141 UTC [27132] WARNING: there is no transaction in progress
2026-08-12 02:33:04,066 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-12 02:33:04,067 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbep4M4/build/debug/bin/postgresql.jar:/tmp/dist-test-taskbep4M4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.103.20:43159/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-12 02:33:04,364 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-12 02:33:04,365 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbep4M4/build/debug/bin/postgresql.jar:/tmp/dist-test-taskbep4M4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.103.20:43159/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-12 02:33:04,668 [I] Table x_portal_user already exists in database 'ranger'
2026-08-12 02:33:04,668 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbep4M4/build/debug/bin/postgresql.jar:/tmp/dist-test-taskbep4M4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.103.20:43159/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-12 02:33:04,958 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-12 02:33:04,959 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbep4M4/build/debug/bin/postgresql.jar:/tmp/dist-test-taskbep4M4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.103.20:43159/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-12 02:33:05,248 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbep4M4/build/debug/bin/postgresql.jar:/tmp/dist-test-taskbep4M4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.103.20:43159/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-12 02:33:05,546 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbep4M4/build/debug/bin/postgresql.jar:/tmp/dist-test-taskbep4M4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.103.20:43159/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-2w3w.c.gcp-upstream.internal';"
2026-08-12 02:33:05,822 [I] CORE_DB_SCHEMA import status has been updated
2026-08-12 02:33:05,822 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbep4M4/build/debug/bin/postgresql.jar:/tmp/dist-test-taskbep4M4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.103.20:43159/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-12 02:33:06,088 [I] DB_PATCHES have already been applied
I20260812 02:33:06.094350 14748 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501898713699-14748-0/minicluster-data/ranger-admin:/tmp/dist-test-taskbep4M4/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskbep4M4/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskbep4M4/build/debug/bin/postgresql.jar:/tmp/dist-test-taskbep4M4/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260812 02:33:06.094465 14748 mini_ranger.cc:193] Using host: 127.14.103.20
I20260812 02:33:06.099368 14748 mini_ranger.cc:227] Ranger admin URL: http://127.14.103.20:36903
Aug 12, 2026 2:33:06 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 12, 2026 2:33:06 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 12, 2026 2:33:06 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501898713699-14748-0/minicluster-data/ranger-admin/ews/webapp
Aug 12, 2026 2:33:06 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501898713699-14748-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 12, 2026 2:33:06 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501898713699-14748-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 12, 2026 2:33:07 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501898713699-14748-0/minicluster-data/ranger-admin/ews/webapp].
Aug 12, 2026 2:33:07 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260812 02:33:07.361037 14748 mini_ranger.cc:160] Time spent starting Ranger: real 10.347s user 0.007s sys 0.008s
Aug 12, 2026 2:33:30 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 12, 2026 2:33:30 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Aug 12, 2026 2:33:30 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 12, 2026 2:33:30 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 12, 2026 2:33:30 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 12, 2026 2:33:30 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 12, 2026 2:33:30 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 12, 2026 2:33:30 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 12, 2026 2:33:30 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 12, 2026 2:33:30 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 12, 2026 2:33:30 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 12, 2026 2:33:30 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 12, 2026 2:33:30 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 12, 2026 2:33:30 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 12, 2026 2:33:30 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 12, 2026 2:33:30 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 12, 2026 2:33:30 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 12, 2026 2:33:31 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260812 02:33:31.955776 14748 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.14.103.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.14.103.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.14.103.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501898713699-14748-0/krb5kdc/rangerkms_127.14.103.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.14.103.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501898713699-14748-0/krb5kdc/rangerkms_127.14.103.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.14.103.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.14.103.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.103.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501898713699-14748-0/krb5kdc/HTTP_127.14.103.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.14.103.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501898713699-14748-0/krb5kdc/HTTP_127.14.103.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 12 02:33:32 dist-test-slave-2w3w krb5kdc[26875](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502012, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260812 02:33:32.073544 14748 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260812 02:33:32.086174 14748 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260812 02:33:33.527499 14748 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260812 02:33:33.527623 14748 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501898713699-14748-0/minicluster-data/ranger-kms
I20260812 02:33:33.527743 14748 mini_ranger_kms.cc:227] Using postgres at 127.14.103.20:43159
2026-08-12 02:33:33,826 [I] DB FLAVOR :POSTGRES
2026-08-12 02:33:33,827 [I] --------- Verifying Ranger DB connection ---------
2026-08-12 02:33:33,827 [I] Checking connection
2026-08-12 02:33:33,827 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbep4M4/build/debug/bin/postgresql.jar:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501898713699-14748-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.103.20:43159/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-12 02:33:34,163 [I] connection success
2026-08-12 02:33:34,163 [I] --------- Verifying Ranger DB tables ---------
2026-08-12 02:33:34,163 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-12 02:33:34,163 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbep4M4/build/debug/bin/postgresql.jar:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501898713699-14748-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.103.20:43159/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-12 02:33:34,468 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-12 02:33:34,469 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-12 02:33:34,469 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-12 02:33:34,469 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbep4M4/build/debug/bin/postgresql.jar:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501898713699-14748-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.103.20:43159/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501898713699-14748-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-12 02:33:34,787 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260812 02:33:35.451122 14748 mini_ranger_kms.cc:310] Created kms service
I20260812 02:33:35.742385 14748 mini_ranger_kms.cc:326] Created kudu user
I20260812 02:33:35.792121 14748 mini_ranger_kms.cc:343] Created rangerkms user
Aug 12 02:33:35 dist-test-slave-2w3w krb5kdc[26875](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502012, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.14.103.20@KRBTEST.COM
I20260812 02:33:36.277200 14748 mini_ranger_kms.cc:384] Added ranger policy
I20260812 02:33:36.277596 14748 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501898713699-14748-0/minicluster-data/ranger-kms:/tmp/dist-test-taskbep4M4/build/debug/bin/postgresql.jar:/tmp/dist-test-taskbep4M4/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskbep4M4/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260812 02:33:36.277621 14748 mini_ranger_kms.cc:240] Using host: 127.14.103.20
I20260812 02:33:36.283325 14748 mini_ranger_kms.cc:276] Ranger KMS PID: 27339
I20260812 02:33:36.283504 14748 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.14.103.20:40897
Aug 12, 2026 2:33:37 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 12, 2026 2:33:37 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 12, 2026 2:33:37 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501898713699-14748-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 12, 2026 2:33:37 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501898713699-14748-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 12, 2026 2:33:37 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501898713699-14748-0/minicluster-data/ranger-kms/ews/webapp].
Aug 12, 2026 2:33:37 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260812 02:33:38.754065 14748 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 6.681s user 0.008s sys 0.269s
I20260812 02:33:38.754208 14748 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260812 02:33:38.754315 14748 mini_ranger_kms.cc:401] 127.14.103.20:40897/kms/v1/keys
Aug 12 02:33:38 dist-test-slave-2w3w krb5kdc[26875](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786502018, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.14.103.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Aug 12 02:33:39 dist-test-slave-2w3w krb5kdc[26875](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502018, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.14.103.20@KRBTEST.COM for HTTP/127.14.103.20@KRBTEST.COM
Aug 12 02:33:39 dist-test-slave-2w3w krb5kdc[26875](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502018, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.14.103.20@KRBTEST.COM for HTTP/127.14.103.20@KRBTEST.COM
Aug 12 02:33:42 dist-test-slave-2w3w krb5kdc[26875](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502022, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.14.103.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.103.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.103.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.62.keytab.
Entry for principal kudu/127.14.103.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.62.keytab.
I20260812 02:33:42.363169 14748 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbep4M4/build/debug/bin/kudu
/tmp/dist-test-taskbep4M4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501898713699-14748-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501898713699-14748-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501898713699-14748-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501898713699-14748-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.103.62:41473
--webserver_interface=127.14.103.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.62.keytab
--principal=kudu/127.14.103.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.103.20:34081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.103.62:41473
--ranger_config_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501898713699-14748-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.103.20:40897/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501898713699-14748-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501898713699-14748-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501898713699-14748-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:33:42.517163 27394 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:33:42.517532 27394 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:33:42.517581 27394 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:33:42.522655 27394 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:33:42.522750 27394 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:33:42.522780 27394 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:33:42.522799 27394 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:33:42.522814 27394 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 02:33:42.529839 27394 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.103.20:34081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501898713699-14748-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501898713699-14748-0/minicluster-data/master-0/wal
--ranger_kms_url=127.14.103.20:40897/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.14.103.62:41473
--ranger_config_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501898713699-14748-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.62.keytab
--principal=kudu/127.14.103.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.103.62:41473
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501898713699-14748-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.103.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27394
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501898713699-14748-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:33:42.531252 27394 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:33:42.532369 27394 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:33:42.542624 27400 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:33:42.542888 27394 server_base.cc:1034] running on GCE node
W20260812 02:33:42.542750 27401 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:33:42.543030 27403 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:33:42.543913 27394 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:33:42.545207 27394 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:33:42.546452 27394 hybrid_clock.cc:648] HybridClock initialized: now 1786502022546433 us; error 47 us; skew 500 ppm
Aug 12 02:33:42 dist-test-slave-2w3w krb5kdc[26875](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502022, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:33:42.550153 27394 init.cc:377] Logged in from keytab as kudu/127.14.103.62@KRBTEST.COM (short username kudu)
I20260812 02:33:42.551403 27394 webserver.cc:460] Webserver started at http://127.14.103.62:34429/ using document root <none> and password file <none>
I20260812 02:33:42.552006 27394 fs_manager.cc:359] Metadata directory not provided
I20260812 02:33:42.552094 27394 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:33:42.552342 27394 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 12 02:33:42 dist-test-slave-2w3w krb5kdc[26875](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502022, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.62@KRBTEST.COM for HTTP/127.14.103.20@KRBTEST.COM
I20260812 02:33:42.614655 27394 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501898713699-14748-0/minicluster-data/master-0/data/instance:
uuid: "5801c57365684accaafb51587361b050"
format_stamp: "Formatted at 2026-08-12 02:33:42 on dist-test-slave-2w3w"
server_key: "f265bd9c0f5c12dd1299e93b0cfec398"
server_key_iv: "dd9cd558b30f4120e6ee559921bee0e5"
server_key_version: "kuduclusterkey@0"
I20260812 02:33:42.615660 27394 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501898713699-14748-0/minicluster-data/master-0/wal/instance:
uuid: "5801c57365684accaafb51587361b050"
format_stamp: "Formatted at 2026-08-12 02:33:42 on dist-test-slave-2w3w"
server_key: "f265bd9c0f5c12dd1299e93b0cfec398"
server_key_iv: "dd9cd558b30f4120e6ee559921bee0e5"
server_key_version: "kuduclusterkey@0"
I20260812 02:33:42.621503 27394 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260812 02:33:42.641238 27411 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:33:42.643049 27394 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260812 02:33:42.643226 27394 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501898713699-14748-0/minicluster-data/master-0/data,/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501898713699-14748-0/minicluster-data/master-0/wal
uuid: "5801c57365684accaafb51587361b050"
format_stamp: "Formatted at 2026-08-12 02:33:42 on dist-test-slave-2w3w"
server_key: "f265bd9c0f5c12dd1299e93b0cfec398"
server_key_iv: "dd9cd558b30f4120e6ee559921bee0e5"
server_key_version: "kuduclusterkey@0"
I20260812 02:33:42.643388 27394 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501898713699-14748-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501898713699-14748-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501898713699-14748-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:33:42.674366 27394 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:33:42.678570 27394 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:33:42.678797 27394 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:33:42.691228 27394 rpc_server.cc:307] RPC server started. Bound to: 127.14.103.62:41473
I20260812 02:33:42.691323 27470 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.103.62:41473 every 8 connection(s)
I20260812 02:33:42.692569 27394 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501898713699-14748-0/minicluster-data/master-0/data/info.pb
I20260812 02:33:42.694872 14748 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbep4M4/build/debug/bin/kudu as pid 27394
I20260812 02:33:42.695052 14748 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501898713699-14748-0/minicluster-data/master-0/wal/instance
I20260812 02:33:42.695369 14748 external_mini_cluster.cc:1428] Setting key d84f97b6257638f738b3c31126d4e9b2
I20260812 02:33:42.696285 27471 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:33:42.704545 27471 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5801c57365684accaafb51587361b050: Bootstrap starting.
Aug 12 02:33:42 dist-test-slave-2w3w krb5kdc[26875](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502022, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.103.62@KRBTEST.COM
I20260812 02:33:42.707762 27471 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5801c57365684accaafb51587361b050: Neither blocks nor log segments found. Creating new log.
I20260812 02:33:42.708835 27471 log.cc:824] T 00000000000000000000000000000000 P 5801c57365684accaafb51587361b050: Log is configured to *not* fsync() on all Append() calls
I20260812 02:33:42.711796 27471 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5801c57365684accaafb51587361b050: No bootstrap required, opened a new log
I20260812 02:33:42.714713 27475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:33:42.696925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48930 (local address 127.14.103.62:41473)
0812 02:33:42.697482 (+ 557us) server_negotiation.cc:206] Beginning negotiation
0812 02:33:42.697501 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0812 02:33:42.697548 (+ 47us) server_negotiation.cc:407] Connection header received
0812 02:33:42.698470 (+ 922us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:33:42.698494 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:33:42.699010 (+ 516us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:33:42.699426 (+ 416us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:33:42.700737 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:33:42.702335 (+ 1598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:33:42.703706 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:33:42.704429 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:33:42.704604 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:33:42.708065 (+ 3461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:33:42.708075 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:33:42.708100 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:33:42.708145 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:33:42.710991 (+ 2846us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:33:42.711634 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:33:42.711639 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:33:42.711646 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:33:42.711786 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:33:42.712112 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:33:42.712116 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:33:42.712118 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:33:42.712798 (+ 680us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:33:42.713010 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0812 02:33:42.713326 (+ 316us) server_negotiation.cc:299] Negotiation successful
0812 02:33:42.713636 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":176,"threads_started":1}
I20260812 02:33:42.716578 27471 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5801c57365684accaafb51587361b050 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5801c57365684accaafb51587361b050" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 41473 } }
I20260812 02:33:42.716876 27471 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5801c57365684accaafb51587361b050 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:33:42.716960 27471 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5801c57365684accaafb51587361b050 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5801c57365684accaafb51587361b050, State: Initialized, Role: FOLLOWER
I20260812 02:33:42.717456 27471 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5801c57365684accaafb51587361b050 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5801c57365684accaafb51587361b050" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 41473 } }
I20260812 02:33:42.717623 27471 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5801c57365684accaafb51587361b050 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 02:33:42.717710 27471 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5801c57365684accaafb51587361b050 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 02:33:42.717816 27471 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5801c57365684accaafb51587361b050 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:33:42.719063 27471 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5801c57365684accaafb51587361b050 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5801c57365684accaafb51587361b050" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 41473 } }
I20260812 02:33:42.719467 27471 leader_election.cc:302] T 00000000000000000000000000000000 P 5801c57365684accaafb51587361b050 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5801c57365684accaafb51587361b050; no voters:
I20260812 02:33:42.719862 27471 leader_election.cc:288] T 00000000000000000000000000000000 P 5801c57365684accaafb51587361b050 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 02:33:42.720139 27477 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5801c57365684accaafb51587361b050 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:33:42.720417 27477 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5801c57365684accaafb51587361b050 [term 1 LEADER]: Becoming Leader. State: Replica: 5801c57365684accaafb51587361b050, State: Running, Role: LEADER
I20260812 02:33:42.721390 27477 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5801c57365684accaafb51587361b050 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5801c57365684accaafb51587361b050" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 41473 } }
I20260812 02:33:42.721575 27471 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5801c57365684accaafb51587361b050 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:33:42.724478 27479 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5801c57365684accaafb51587361b050 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5801c57365684accaafb51587361b050. Latest consensus state: current_term: 1 leader_uuid: "5801c57365684accaafb51587361b050" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5801c57365684accaafb51587361b050" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 41473 } } }
I20260812 02:33:42.724649 27479 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5801c57365684accaafb51587361b050 [sys.catalog]: This master's current role is: LEADER
I20260812 02:33:42.724748 27471 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501898713699-14748-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260812 02:33:42.724978 27478 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5801c57365684accaafb51587361b050 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5801c57365684accaafb51587361b050" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5801c57365684accaafb51587361b050" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 41473 } } }
I20260812 02:33:42.725060 27478 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5801c57365684accaafb51587361b050 [sys.catalog]: This master's current role is: LEADER
I20260812 02:33:42.726619 27494 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 02:33:42.733835 27494 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 02:33:42.744055 27494 catalog_manager.cc:1347] Generated new cluster ID: b952765b48064358894820ed0ab65a17
I20260812 02:33:42.744134 27494 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 02:33:42.755929 27494 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 02:33:42.757440 27494 catalog_manager.cc:1504] Loading token signing keys...
I20260812 02:33:42.769702 27494 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5801c57365684accaafb51587361b050: Generated new TSK 0
I20260812 02:33:42.771049 27494 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.103.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.103.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.103.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.1.keytab.
Entry for principal kudu/127.14.103.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.1.keytab.
I20260812 02:33:43.583741 14748 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbep4M4/build/debug/bin/kudu
/tmp/dist-test-taskbep4M4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501898713699-14748-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501898713699-14748-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501898713699-14748-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501898713699-14748-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.103.20:40897/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.103.1:0
--local_ip_for_outbound_sockets=127.14.103.1
--webserver_interface=127.14.103.1
--webserver_port=0
--tserver_master_addrs=127.14.103.62:41473
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.1.keytab
--principal=kudu/127.14.103.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.103.20:34081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501898713699-14748-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501898713699-14748-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501898713699-14748-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:33:43.725921 27524 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:33:43.726269 27524 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:33:43.726389 27524 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:33:43.731060 27524 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:33:43.731158 27524 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:33:43.731254 27524 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.103.1
I20260812 02:33:43.737831 27524 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.103.20:34081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501898713699-14748-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501898713699-14748-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.14.103.20:40897/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.1.keytab
--principal=kudu/127.14.103.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.103.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501898713699-14748-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.103.1
--webserver_port=0
--tserver_master_addrs=127.14.103.62:41473
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27524
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.103.1
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501898713699-14748-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:33:43.739348 27524 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:33:43.740723 27524 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:33:43.754815 27533 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:33:43.754900 27534 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:33:43.754873 27536 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:33:43.755259 27524 server_base.cc:1034] running on GCE node
I20260812 02:33:43.755730 27524 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:33:43.756429 27524 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:33:43.757650 27524 hybrid_clock.cc:648] HybridClock initialized: now 1786502023757616 us; error 49 us; skew 500 ppm
Aug 12 02:33:43 dist-test-slave-2w3w krb5kdc[26875](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502023, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:33:43.761615 27524 init.cc:377] Logged in from keytab as kudu/127.14.103.1@KRBTEST.COM (short username kudu)
I20260812 02:33:43.762729 27524 webserver.cc:460] Webserver started at http://127.14.103.1:38519/ using document root <none> and password file <none>
I20260812 02:33:43.763396 27524 fs_manager.cc:359] Metadata directory not provided
I20260812 02:33:43.763501 27524 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:33:43.763747 27524 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 12 02:33:43 dist-test-slave-2w3w krb5kdc[26875](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502023, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.1@KRBTEST.COM for HTTP/127.14.103.20@KRBTEST.COM
I20260812 02:33:43.784022 27524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501898713699-14748-0/minicluster-data/ts-0/data/instance:
uuid: "4675861452d9490f9521569aaa091fa8"
format_stamp: "Formatted at 2026-08-12 02:33:43 on dist-test-slave-2w3w"
server_key: "02537f6d35dee69396304fe3a34767fa"
server_key_iv: "9f31e12ef528dfa164ba645d267cb643"
server_key_version: "kuduclusterkey@0"
I20260812 02:33:43.784873 27524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501898713699-14748-0/minicluster-data/ts-0/wal/instance:
uuid: "4675861452d9490f9521569aaa091fa8"
format_stamp: "Formatted at 2026-08-12 02:33:43 on dist-test-slave-2w3w"
server_key: "02537f6d35dee69396304fe3a34767fa"
server_key_iv: "9f31e12ef528dfa164ba645d267cb643"
server_key_version: "kuduclusterkey@0"
I20260812 02:33:43.790490 27524 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260812 02:33:43.811286 27546 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:33:43.813079 27524 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260812 02:33:43.813239 27524 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501898713699-14748-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501898713699-14748-0/minicluster-data/ts-0/wal
uuid: "4675861452d9490f9521569aaa091fa8"
format_stamp: "Formatted at 2026-08-12 02:33:43 on dist-test-slave-2w3w"
server_key: "02537f6d35dee69396304fe3a34767fa"
server_key_iv: "9f31e12ef528dfa164ba645d267cb643"
server_key_version: "kuduclusterkey@0"
I20260812 02:33:43.813390 27524 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501898713699-14748-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501898713699-14748-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501898713699-14748-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:33:43.857388 27524 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:33:43.861392 27524 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:33:43.861683 27524 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:33:43.862473 27524 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:33:43.863669 27524 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:33:43.863790 27524 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:33:43.863929 27524 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:33:43.864022 27524 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:33:43.877792 27524 rpc_server.cc:307] RPC server started. Bound to: 127.14.103.1:34951
I20260812 02:33:43.879384 27675 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.103.1:34951 every 8 connection(s)
I20260812 02:33:43.880728 27524 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501898713699-14748-0/minicluster-data/ts-0/data/info.pb
I20260812 02:33:43.890389 14748 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbep4M4/build/debug/bin/kudu as pid 27524
I20260812 02:33:43.890530 14748 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501898713699-14748-0/minicluster-data/ts-0/wal/instance
I20260812 02:33:43.890878 14748 external_mini_cluster.cc:1428] Setting key 287955471ff4ccb9bc1a65c9896d4dd0
Aug 12 02:33:43 dist-test-slave-2w3w krb5kdc[26875](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502023, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.1@KRBTEST.COM for kudu/127.14.103.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.14.103.2@KRBTEST.COM; defaulting to no policy
I20260812 02:33:43.931062 27680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:33:43.884147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.1:59115 (local address 127.14.103.62:41473)
0812 02:33:43.902028 (+ 17881us) server_negotiation.cc:206] Beginning negotiation
0812 02:33:43.902033 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:33:43.902056 (+ 23us) server_negotiation.cc:407] Connection header received
0812 02:33:43.902112 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:33:43.902116 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:33:43.902175 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:33:43.902306 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:33:43.904188 (+ 1882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:33:43.905094 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:33:43.914015 (+ 8921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:33:43.914446 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:33:43.914629 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:33:43.922355 (+ 7726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:33:43.922360 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:33:43.922363 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:33:43.922399 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:33:43.924470 (+ 2071us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:33:43.927362 (+ 2892us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:33:43.927366 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:33:43.927368 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:33:43.927540 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:33:43.928042 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:33:43.928047 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:33:43.928051 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:33:43.928304 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:33:43.928390 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0812 02:33:43.930676 (+ 2286us) server_negotiation.cc:299] Negotiation successful
0812 02:33:43.930869 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17764,"thread_start_us":93,"threads_started":1}
I20260812 02:33:43.932469 27681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:33:43.884600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:41473 (local address 127.14.103.1:59115)
0812 02:33:43.893116 (+ 8516us) negotiation.cc:107] Waiting for socket to connect
0812 02:33:43.893208 (+ 92us) client_negotiation.cc:174] Beginning negotiation
0812 02:33:43.894364 (+ 1156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:33:43.902651 (+ 8287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:33:43.902665 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:33:43.903150 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:33:43.903938 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:33:43.903964 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:33:43.905311 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:33:43.905316 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:33:43.907284 (+ 1968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:33:43.907300 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:33:43.916433 (+ 9133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:33:43.916439 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:33:43.916742 (+ 303us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:33:43.918517 (+ 1775us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:33:43.918545 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:33:43.922006 (+ 3461us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:33:43.926415 (+ 4409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:33:43.926425 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:33:43.926446 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:33:43.927101 (+ 655us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:33:43.927680 (+ 579us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:33:43.927684 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:33:43.927686 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:33:43.927924 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:33:43.928456 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:33:43.928464 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:33:43.928844 (+ 380us) client_negotiation.cc:769] Sending connection context
0812 02:33:43.929056 (+ 212us) client_negotiation.cc:240] Negotiation successful
0812 02:33:43.929448 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8196,"thread_start_us":157,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.103.2@KRBTEST.COM" created.
I20260812 02:33:43.934433 27676 heartbeater.cc:347] Connected to a master server at 127.14.103.62:41473
I20260812 02:33:43.934716 27676 heartbeater.cc:464] Registering TS with master...
I20260812 02:33:43.935539 27676 heartbeater.cc:511] Master 127.14.103.62:41473 requested a full tablet report, sending...
I20260812 02:33:43.937438 27428 ts_manager.cc:194] Registered new tserver with Master: 4675861452d9490f9521569aaa091fa8 (127.14.103.1:34951)
I20260812 02:33:43.939139 27428 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.103.1@KRBTEST.COM'} at 127.14.103.1:59115
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.103.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.2.keytab.
Entry for principal kudu/127.14.103.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.2.keytab.
I20260812 02:33:43.971359 14748 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbep4M4/build/debug/bin/kudu
/tmp/dist-test-taskbep4M4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501898713699-14748-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501898713699-14748-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501898713699-14748-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501898713699-14748-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.103.20:40897/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.103.2:0
--local_ip_for_outbound_sockets=127.14.103.2
--webserver_interface=127.14.103.2
--webserver_port=0
--tserver_master_addrs=127.14.103.62:41473
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.2.keytab
--principal=kudu/127.14.103.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.103.20:34081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501898713699-14748-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501898713699-14748-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501898713699-14748-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 02:33:44 dist-test-slave-2w3w krb5kdc[26875](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786502024, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260812 02:33:44.152522 27687 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:33:44.152988 27687 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:33:44.153074 27687 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:33:44.160524 27687 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:33:44.160660 27687 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:33:44.160760 27687 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.103.2
I20260812 02:33:44.170585 27687 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.103.20:34081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501898713699-14748-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501898713699-14748-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.14.103.20:40897/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.2.keytab
--principal=kudu/127.14.103.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.103.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501898713699-14748-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.103.2
--webserver_port=0
--tserver_master_addrs=127.14.103.62:41473
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27687
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.103.2
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501898713699-14748-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:33:44.172185 27687 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:33:44.173604 27687 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:33:44.186102 27694 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:33:44.186219 27695 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:33:44.187054 27698 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:33:44.187336 27687 server_base.cc:1034] running on GCE node
I20260812 02:33:44.188010 27687 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:33:44.188875 27687 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:33:44.190090 27687 hybrid_clock.cc:648] HybridClock initialized: now 1786502024190041 us; error 61 us; skew 500 ppm
Aug 12 02:33:44 dist-test-slave-2w3w krb5kdc[26875](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502024, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:33:44.199338 27687 init.cc:377] Logged in from keytab as kudu/127.14.103.2@KRBTEST.COM (short username kudu)
I20260812 02:33:44.200749 27687 webserver.cc:460] Webserver started at http://127.14.103.2:34041/ using document root <none> and password file <none>
I20260812 02:33:44.201668 27687 fs_manager.cc:359] Metadata directory not provided
I20260812 02:33:44.201797 27687 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:33:44.202189 27687 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 12 02:33:44 dist-test-slave-2w3w krb5kdc[26875](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502024, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.2@KRBTEST.COM for HTTP/127.14.103.20@KRBTEST.COM
I20260812 02:33:44.221123 27687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501898713699-14748-0/minicluster-data/ts-1/data/instance:
uuid: "e49da65941fe480583c6ea9e169aa8a8"
format_stamp: "Formatted at 2026-08-12 02:33:44 on dist-test-slave-2w3w"
server_key: "05a4334adfa4a0e76220c791980d33a0"
server_key_iv: "cc211154a95b26d62d5fb97911024b77"
server_key_version: "kuduclusterkey@0"
I20260812 02:33:44.222123 27687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501898713699-14748-0/minicluster-data/ts-1/wal/instance:
uuid: "e49da65941fe480583c6ea9e169aa8a8"
format_stamp: "Formatted at 2026-08-12 02:33:44 on dist-test-slave-2w3w"
server_key: "05a4334adfa4a0e76220c791980d33a0"
server_key_iv: "cc211154a95b26d62d5fb97911024b77"
server_key_version: "kuduclusterkey@0"
I20260812 02:33:44.231609 27687 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.005s sys 0.000s
I20260812 02:33:44.262605 27706 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:33:44.264465 27687 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260812 02:33:44.264639 27687 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501898713699-14748-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501898713699-14748-0/minicluster-data/ts-1/wal
uuid: "e49da65941fe480583c6ea9e169aa8a8"
format_stamp: "Formatted at 2026-08-12 02:33:44 on dist-test-slave-2w3w"
server_key: "05a4334adfa4a0e76220c791980d33a0"
server_key_iv: "cc211154a95b26d62d5fb97911024b77"
server_key_version: "kuduclusterkey@0"
I20260812 02:33:44.264808 27687 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501898713699-14748-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501898713699-14748-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501898713699-14748-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:33:44.326592 27687 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:33:44.331247 27687 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:33:44.331476 27687 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:33:44.332407 27687 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:33:44.333702 27687 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:33:44.333775 27687 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:33:44.333841 27687 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:33:44.333874 27687 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:33:44.364038 27687 rpc_server.cc:307] RPC server started. Bound to: 127.14.103.2:43407
I20260812 02:33:44.364076 27837 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.103.2:43407 every 8 connection(s)
I20260812 02:33:44.365602 27687 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501898713699-14748-0/minicluster-data/ts-1/data/info.pb
I20260812 02:33:44.383322 14748 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbep4M4/build/debug/bin/kudu as pid 27687
I20260812 02:33:44.383471 14748 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501898713699-14748-0/minicluster-data/ts-1/wal/instance
I20260812 02:33:44.383777 14748 external_mini_cluster.cc:1428] Setting key 2f8e1960f58e8acd480aedbbb227198a
Aug 12 02:33:44 dist-test-slave-2w3w krb5kdc[26875](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502024, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.2@KRBTEST.COM for kudu/127.14.103.62@KRBTEST.COM
I20260812 02:33:44.394158 27680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:33:44.371252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.2:50989 (local address 127.14.103.62:41473)
0812 02:33:44.371527 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0812 02:33:44.371533 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 02:33:44.372074 (+ 541us) server_negotiation.cc:407] Connection header received
0812 02:33:44.373249 (+ 1175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:33:44.373253 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:33:44.373317 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:33:44.373449 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:33:44.374848 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:33:44.375813 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:33:44.378377 (+ 2564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:33:44.378746 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:33:44.378870 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:33:44.388324 (+ 9454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:33:44.388329 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:33:44.388332 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:33:44.388371 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:33:44.391174 (+ 2803us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:33:44.391984 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:33:44.391989 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:33:44.391993 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:33:44.392113 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:33:44.392601 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:33:44.392608 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:33:44.392610 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:33:44.392905 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:33:44.393035 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0812 02:33:44.393752 (+ 717us) server_negotiation.cc:299] Negotiation successful
0812 02:33:44.393926 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170}
I20260812 02:33:44.395336 27840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:33:44.369793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:41473 (local address 127.14.103.2:50989)
0812 02:33:44.371871 (+ 2078us) negotiation.cc:107] Waiting for socket to connect
0812 02:33:44.371926 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0812 02:33:44.372983 (+ 1057us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:33:44.373666 (+ 683us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:33:44.373675 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:33:44.374015 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:33:44.374646 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:33:44.374662 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:33:44.376979 (+ 2317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:33:44.376984 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:33:44.378188 (+ 1204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:33:44.378203 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:33:44.382363 (+ 4160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:33:44.382372 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:33:44.382852 (+ 480us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:33:44.383970 (+ 1118us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:33:44.383996 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:33:44.385888 (+ 1892us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:33:44.391363 (+ 5475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:33:44.391372 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:33:44.391405 (+ 33us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:33:44.391854 (+ 449us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:33:44.392243 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:33:44.392246 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:33:44.392248 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:33:44.392486 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:33:44.393033 (+ 547us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:33:44.393041 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:33:44.393440 (+ 399us) client_negotiation.cc:769] Sending connection context
0812 02:33:44.393694 (+ 254us) client_negotiation.cc:240] Negotiation successful
0812 02:33:44.394100 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1859,"thread_start_us":202,"threads_started":1}
I20260812 02:33:44.396883 27838 heartbeater.cc:347] Connected to a master server at 127.14.103.62:41473
I20260812 02:33:44.397451 27838 heartbeater.cc:464] Registering TS with master...
I20260812 02:33:44.398761 27838 heartbeater.cc:511] Master 127.14.103.62:41473 requested a full tablet report, sending...
I20260812 02:33:44.400789 27428 ts_manager.cc:194] Registered new tserver with Master: e49da65941fe480583c6ea9e169aa8a8 (127.14.103.2:43407)
I20260812 02:33:44.401845 27428 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.103.2@KRBTEST.COM'} at 127.14.103.2:50989
WARNING: no policy specified for kudu/127.14.103.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.103.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.103.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.3.keytab.
Entry for principal kudu/127.14.103.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.3.keytab.
I20260812 02:33:44.444187 14748 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbep4M4/build/debug/bin/kudu
/tmp/dist-test-taskbep4M4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501898713699-14748-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501898713699-14748-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501898713699-14748-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501898713699-14748-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.103.20:40897/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.103.3:0
--local_ip_for_outbound_sockets=127.14.103.3
--webserver_interface=127.14.103.3
--webserver_port=0
--tserver_master_addrs=127.14.103.62:41473
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.3.keytab
--principal=kudu/127.14.103.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.103.20:34081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501898713699-14748-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501898713699-14748-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501898713699-14748-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:33:44.667297 27843 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:33:44.667799 27843 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:33:44.667917 27843 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:33:44.674474 27843 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:33:44.674661 27843 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:33:44.674815 27843 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.103.3
I20260812 02:33:44.684175 27843 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.103.20:34081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501898713699-14748-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501898713699-14748-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.14.103.20:40897/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.3.keytab
--principal=kudu/127.14.103.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.103.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501898713699-14748-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.103.3
--webserver_port=0
--tserver_master_addrs=127.14.103.62:41473
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27843
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.103.3
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501898713699-14748-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:33:44.686327 27843 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:33:44.687906 27843 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:33:44.703019 27849 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:33:44.703817 27843 server_base.cc:1034] running on GCE node
W20260812 02:33:44.704448 27848 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:33:44.719267 27851 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:33:44.720194 27843 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:33:44.721207 27843 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:33:44.736752 27843 hybrid_clock.cc:648] HybridClock initialized: now 1786502024736627 us; error 148 us; skew 500 ppm
Aug 12 02:33:44 dist-test-slave-2w3w krb5kdc[26875](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502024, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:33:44.741255 27843 init.cc:377] Logged in from keytab as kudu/127.14.103.3@KRBTEST.COM (short username kudu)
I20260812 02:33:44.742574 27843 webserver.cc:460] Webserver started at http://127.14.103.3:44539/ using document root <none> and password file <none>
Aug 12 02:33:44 dist-test-slave-2w3w krb5kdc[26875](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502024, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.62@KRBTEST.COM for HTTP/127.14.103.20@KRBTEST.COM
I20260812 02:33:44.744176 27843 fs_manager.cc:359] Metadata directory not provided
I20260812 02:33:44.744323 27843 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:33:44.744602 27843 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 12 02:33:44 dist-test-slave-2w3w krb5kdc[26875](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502024, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.3@KRBTEST.COM for HTTP/127.14.103.20@KRBTEST.COM
I20260812 02:33:44.776300 27843 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501898713699-14748-0/minicluster-data/ts-2/data/instance:
uuid: "accfa5b49d7a48e58b1751893eb3ce33"
format_stamp: "Formatted at 2026-08-12 02:33:44 on dist-test-slave-2w3w"
server_key: "138887f225ac70289b8843ffc64d99cc"
server_key_iv: "15ca597ef3e0e68de46707e390884a21"
server_key_version: "kuduclusterkey@0"
I20260812 02:33:44.787766 27843 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501898713699-14748-0/minicluster-data/ts-2/wal/instance:
uuid: "accfa5b49d7a48e58b1751893eb3ce33"
format_stamp: "Formatted at 2026-08-12 02:33:44 on dist-test-slave-2w3w"
server_key: "138887f225ac70289b8843ffc64d99cc"
server_key_iv: "15ca597ef3e0e68de46707e390884a21"
server_key_version: "kuduclusterkey@0"
I20260812 02:33:44.794452 27843 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260812 02:33:44.821776 27858 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:33:44.824124 27843 fs_manager.cc:725] Time spent opening block manager: real 0.013s user 0.002s sys 0.000s
I20260812 02:33:44.824352 27843 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501898713699-14748-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501898713699-14748-0/minicluster-data/ts-2/wal
uuid: "accfa5b49d7a48e58b1751893eb3ce33"
format_stamp: "Formatted at 2026-08-12 02:33:44 on dist-test-slave-2w3w"
server_key: "138887f225ac70289b8843ffc64d99cc"
server_key_iv: "15ca597ef3e0e68de46707e390884a21"
server_key_version: "kuduclusterkey@0"
I20260812 02:33:44.824522 27843 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501898713699-14748-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501898713699-14748-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501898713699-14748-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:33:44.864205 27843 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:33:44.869000 27843 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:33:44.869303 27843 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:33:44.870101 27843 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:33:44.871289 27843 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:33:44.871423 27843 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:33:44.871522 27843 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:33:44.871587 27843 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:33:44.890488 27843 rpc_server.cc:307] RPC server started. Bound to: 127.14.103.3:35951
I20260812 02:33:44.891877 27843 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501898713699-14748-0/minicluster-data/ts-2/data/info.pb
I20260812 02:33:44.895619 14748 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbep4M4/build/debug/bin/kudu as pid 27843
I20260812 02:33:44.895748 14748 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501898713699-14748-0/minicluster-data/ts-2/wal/instance
I20260812 02:33:44.896054 14748 external_mini_cluster.cc:1428] Setting key 39a2add80f865a02b1a269d5ec67b3e6
I20260812 02:33:44.907444 27995 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.103.3:35951 every 8 connection(s)
Aug 12 02:33:44 dist-test-slave-2w3w krb5kdc[26875](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502024, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.3@KRBTEST.COM for kudu/127.14.103.62@KRBTEST.COM
I20260812 02:33:44.924106 27999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:33:44.900848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.3:37571 (local address 127.14.103.62:41473)
0812 02:33:44.907235 (+ 6387us) server_negotiation.cc:206] Beginning negotiation
0812 02:33:44.907241 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 02:33:44.907263 (+ 22us) server_negotiation.cc:407] Connection header received
0812 02:33:44.907321 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:33:44.907324 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:33:44.907385 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:33:44.908716 (+ 1331us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:33:44.909843 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:33:44.910754 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:33:44.912543 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:33:44.912899 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:33:44.913014 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:33:44.916999 (+ 3985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:33:44.917003 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:33:44.917006 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:33:44.917039 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:33:44.919384 (+ 2345us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:33:44.920178 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:33:44.920182 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:33:44.920184 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:33:44.920293 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:33:44.920957 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:33:44.920961 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:33:44.920963 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:33:44.921215 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:33:44.921321 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0812 02:33:44.923661 (+ 2340us) server_negotiation.cc:299] Negotiation successful
0812 02:33:44.923842 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6265,"thread_start_us":96,"threads_started":1}
I20260812 02:33:44.925879 28000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:33:44.901478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:41473 (local address 127.14.103.3:37571)
0812 02:33:44.902257 (+ 779us) negotiation.cc:107] Waiting for socket to connect
0812 02:33:44.902322 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0812 02:33:44.903428 (+ 1106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:33:44.907900 (+ 4472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:33:44.907914 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:33:44.908322 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:33:44.909434 (+ 1112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:33:44.909456 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:33:44.910926 (+ 1470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:33:44.910930 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:33:44.912380 (+ 1450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:33:44.912395 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:33:44.913090 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:33:44.913094 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:33:44.913313 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:33:44.914346 (+ 1033us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:33:44.914398 (+ 52us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:33:44.916694 (+ 2296us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:33:44.919547 (+ 2853us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:33:44.919556 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:33:44.919580 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:33:44.920022 (+ 442us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:33:44.920418 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:33:44.920422 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:33:44.920424 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:33:44.920703 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:33:44.921758 (+ 1055us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:33:44.921767 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:33:44.923272 (+ 1505us) client_negotiation.cc:769] Sending connection context
0812 02:33:44.923552 (+ 280us) client_negotiation.cc:240] Negotiation successful
0812 02:33:44.924011 (+ 459us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":299,"threads_started":1}
I20260812 02:33:44.927542 27996 heartbeater.cc:347] Connected to a master server at 127.14.103.62:41473
I20260812 02:33:44.927838 27996 heartbeater.cc:464] Registering TS with master...
I20260812 02:33:44.928503 27996 heartbeater.cc:511] Master 127.14.103.62:41473 requested a full tablet report, sending...
I20260812 02:33:44.929765 27428 ts_manager.cc:194] Registered new tserver with Master: accfa5b49d7a48e58b1751893eb3ce33 (127.14.103.3:35951)
I20260812 02:33:44.930560 27428 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.103.3@KRBTEST.COM'} at 127.14.103.3:37571
I20260812 02:33:44.935209 14748 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 02:33:44.942730 27676 heartbeater.cc:503] Master 127.14.103.62:41473 was elected leader, sending a full tablet report...
I20260812 02:33:44.951368 27999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:33:44.938437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48932 (local address 127.14.103.62:41473)
0812 02:33:44.938595 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0812 02:33:44.938600 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:33:44.938617 (+ 17us) server_negotiation.cc:407] Connection header received
0812 02:33:44.938670 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:33:44.938673 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:33:44.938720 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:33:44.938826 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:33:44.940053 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:33:44.940846 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:33:44.942723 (+ 1877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:33:44.943226 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:33:44.943357 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:33:44.946334 (+ 2977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:33:44.946339 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:33:44.946342 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:33:44.946378 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:33:44.948763 (+ 2385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:33:44.949577 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:33:44.949590 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:33:44.949592 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:33:44.949797 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:33:44.950214 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:33:44.950219 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:33:44.950221 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:33:44.950476 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:33:44.950595 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0812 02:33:44.951047 (+ 452us) server_negotiation.cc:299] Negotiation successful
0812 02:33:44.951199 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260812 02:33:44.955421 27428 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48932:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 02:33:44.961025 27428 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 02:33:44.996591 28012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:33:44.976642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60422 (local address 127.14.103.3:35951)
0812 02:33:44.983756 (+ 7114us) server_negotiation.cc:206] Beginning negotiation
0812 02:33:44.983769 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0812 02:33:44.988844 (+ 5075us) server_negotiation.cc:407] Connection header received
0812 02:33:44.989004 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:33:44.989011 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:33:44.989281 (+ 270us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:33:44.989477 (+ 196us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:33:44.990272 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:33:44.991544 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:33:44.994000 (+ 2456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:33:44.995109 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:33:44.995275 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:33:44.995475 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0812 02:33:44.996272 (+ 797us) server_negotiation.cc:299] Negotiation successful
0812 02:33:44.996402 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6990,"thread_start_us":174,"threads_started":1}
I20260812 02:33:44.996719 28016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:33:44.982269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.2:43407 (local address 127.0.0.1:37726)
0812 02:33:44.991166 (+ 8897us) negotiation.cc:107] Waiting for socket to connect
0812 02:33:44.991183 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 02:33:44.991292 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:33:44.992142 (+ 850us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:33:44.992146 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:33:44.992170 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:33:44.992275 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:33:44.992282 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:33:44.993752 (+ 1470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:33:44.993755 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:33:44.994897 (+ 1142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:33:44.994908 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:33:44.996229 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:33:44.996232 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:33:44.996323 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:33:44.996344 (+ 21us) client_negotiation.cc:769] Sending connection context
0812 02:33:44.996418 (+ 74us) client_negotiation.cc:240] Negotiation successful
0812 02:33:44.996545 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8807,"thread_start_us":232,"threads_started":1}
I20260812 02:33:44.997222 28018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:33:44.982873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37726 (local address 127.14.103.2:43407)
0812 02:33:44.985566 (+ 2693us) server_negotiation.cc:206] Beginning negotiation
0812 02:33:44.985573 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 02:33:44.991429 (+ 5856us) server_negotiation.cc:407] Connection header received
0812 02:33:44.991538 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:33:44.991545 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:33:44.991815 (+ 270us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:33:44.991986 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:33:44.992429 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:33:44.993592 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:33:44.995101 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:33:44.995923 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:33:44.996031 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:33:44.996159 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0812 02:33:44.996899 (+ 740us) server_negotiation.cc:299] Negotiation successful
0812 02:33:44.997029 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2579,"thread_start_us":130,"threads_started":1}
I20260812 02:33:44.998346 28014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:33:44.976471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.3:35951 (local address 127.0.0.1:60422)
0812 02:33:44.984961 (+ 8490us) negotiation.cc:107] Waiting for socket to connect
0812 02:33:44.985042 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0812 02:33:44.985206 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:33:44.989997 (+ 4791us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:33:44.990000 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:33:44.990023 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:33:44.990147 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:33:44.990152 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:33:44.992442 (+ 2290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:33:44.992445 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:33:44.993749 (+ 1304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:33:44.993758 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:33:44.995846 (+ 2088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:33:44.995850 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:33:44.996037 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:33:44.996089 (+ 52us) client_negotiation.cc:769] Sending connection context
0812 02:33:44.996221 (+ 132us) client_negotiation.cc:240] Negotiation successful
0812 02:33:44.998208 (+ 1987us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8227,"thread_start_us":229,"threads_started":1}
I20260812 02:33:45.000837 27757 tablet_service.cc:1467] Processing CreateTablet for tablet d9303f211bc04a449977a6318b91441a (DEFAULT_TABLE table=test-table [id=b0c95542827e4bd39f48e98fdd0df74c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:33:45.002140 27757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9303f211bc04a449977a6318b91441a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:33:45.002259 27917 tablet_service.cc:1467] Processing CreateTablet for tablet d9303f211bc04a449977a6318b91441a (DEFAULT_TABLE table=test-table [id=b0c95542827e4bd39f48e98fdd0df74c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:33:45.003480 27917 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9303f211bc04a449977a6318b91441a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:33:45.016422 28015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:33:44.980285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45906 (local address 127.14.103.1:34951)
0812 02:33:44.983569 (+ 3284us) server_negotiation.cc:206] Beginning negotiation
0812 02:33:44.984529 (+ 960us) server_negotiation.cc:399] Waiting for connection header
0812 02:33:44.986055 (+ 1526us) server_negotiation.cc:407] Connection header received
0812 02:33:44.986158 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:33:44.986165 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:33:44.986442 (+ 277us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:33:44.986607 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:33:44.988483 (+ 1876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:33:44.989831 (+ 1348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:33:44.994510 (+ 4679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:33:44.995522 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:33:44.995635 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:33:44.995769 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0812 02:33:45.016092 (+ 20323us) server_negotiation.cc:299] Negotiation successful
0812 02:33:45.016251 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3148,"thread_start_us":87,"threads_started":1}
I20260812 02:33:44.999075 28011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:33:44.979706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.1:34951 (local address 127.0.0.1:45906)
0812 02:33:44.985797 (+ 6091us) negotiation.cc:107] Waiting for socket to connect
0812 02:33:44.985812 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 02:33:44.985936 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:33:44.986804 (+ 868us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:33:44.986810 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:33:44.986835 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:33:44.988312 (+ 1477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:33:44.988321 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:33:44.990268 (+ 1947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:33:44.990271 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:33:44.994338 (+ 4067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:33:44.994348 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:33:44.998629 (+ 4281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:33:44.998633 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:33:44.998739 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:33:44.998766 (+ 27us) client_negotiation.cc:769] Sending connection context
0812 02:33:44.998839 (+ 73us) client_negotiation.cc:240] Negotiation successful
0812 02:33:44.998925 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6009,"thread_start_us":211,"threads_started":1}
I20260812 02:33:45.017367 28020 tablet_bootstrap.cc:492] T d9303f211bc04a449977a6318b91441a P e49da65941fe480583c6ea9e169aa8a8: Bootstrap starting.
I20260812 02:33:45.020184 28020 tablet_bootstrap.cc:654] T d9303f211bc04a449977a6318b91441a P e49da65941fe480583c6ea9e169aa8a8: Neither blocks nor log segments found. Creating new log.
I20260812 02:33:45.022140 28020 log.cc:824] T d9303f211bc04a449977a6318b91441a P e49da65941fe480583c6ea9e169aa8a8: Log is configured to *not* fsync() on all Append() calls
I20260812 02:33:45.022055 27579 tablet_service.cc:1467] Processing CreateTablet for tablet d9303f211bc04a449977a6318b91441a (DEFAULT_TABLE table=test-table [id=b0c95542827e4bd39f48e98fdd0df74c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:33:45.025239 28020 tablet_bootstrap.cc:492] T d9303f211bc04a449977a6318b91441a P e49da65941fe480583c6ea9e169aa8a8: No bootstrap required, opened a new log
I20260812 02:33:45.025519 28020 ts_tablet_manager.cc:1397] T d9303f211bc04a449977a6318b91441a P e49da65941fe480583c6ea9e169aa8a8: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260812 02:33:45.025539 27579 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9303f211bc04a449977a6318b91441a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:33:45.030890 28020 raft_consensus.cc:348] T d9303f211bc04a449977a6318b91441a P e49da65941fe480583c6ea9e169aa8a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "accfa5b49d7a48e58b1751893eb3ce33" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 35951 } } peers { permanent_uuid: "4675861452d9490f9521569aaa091fa8" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 34951 } } peers { permanent_uuid: "e49da65941fe480583c6ea9e169aa8a8" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 43407 } }
I20260812 02:33:45.031296 28020 raft_consensus.cc:374] T d9303f211bc04a449977a6318b91441a P e49da65941fe480583c6ea9e169aa8a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:33:45.031363 28020 raft_consensus.cc:729] T d9303f211bc04a449977a6318b91441a P e49da65941fe480583c6ea9e169aa8a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e49da65941fe480583c6ea9e169aa8a8, State: Initialized, Role: FOLLOWER
I20260812 02:33:45.031945 28020 consensus_queue.cc:260] T d9303f211bc04a449977a6318b91441a P e49da65941fe480583c6ea9e169aa8a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "accfa5b49d7a48e58b1751893eb3ce33" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 35951 } } peers { permanent_uuid: "4675861452d9490f9521569aaa091fa8" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 34951 } } peers { permanent_uuid: "e49da65941fe480583c6ea9e169aa8a8" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 43407 } }
I20260812 02:33:45.033197 28020 ts_tablet_manager.cc:1428] T d9303f211bc04a449977a6318b91441a P e49da65941fe480583c6ea9e169aa8a8: Time spent starting tablet: real 0.008s user 0.009s sys 0.000s
I20260812 02:33:45.033661 27838 heartbeater.cc:503] Master 127.14.103.62:41473 was elected leader, sending a full tablet report...
I20260812 02:33:45.037606 28023 tablet_bootstrap.cc:492] T d9303f211bc04a449977a6318b91441a P accfa5b49d7a48e58b1751893eb3ce33: Bootstrap starting.
I20260812 02:33:45.040612 28023 tablet_bootstrap.cc:654] T d9303f211bc04a449977a6318b91441a P accfa5b49d7a48e58b1751893eb3ce33: Neither blocks nor log segments found. Creating new log.
I20260812 02:33:45.041564 28023 log.cc:824] T d9303f211bc04a449977a6318b91441a P accfa5b49d7a48e58b1751893eb3ce33: Log is configured to *not* fsync() on all Append() calls
I20260812 02:33:45.042593 28024 tablet_bootstrap.cc:492] T d9303f211bc04a449977a6318b91441a P 4675861452d9490f9521569aaa091fa8: Bootstrap starting.
I20260812 02:33:45.045258 28024 tablet_bootstrap.cc:654] T d9303f211bc04a449977a6318b91441a P 4675861452d9490f9521569aaa091fa8: Neither blocks nor log segments found. Creating new log.
I20260812 02:33:45.046464 28024 log.cc:824] T d9303f211bc04a449977a6318b91441a P 4675861452d9490f9521569aaa091fa8: Log is configured to *not* fsync() on all Append() calls
I20260812 02:33:45.044304 28023 tablet_bootstrap.cc:492] T d9303f211bc04a449977a6318b91441a P accfa5b49d7a48e58b1751893eb3ce33: No bootstrap required, opened a new log
I20260812 02:33:45.047618 28023 ts_tablet_manager.cc:1397] T d9303f211bc04a449977a6318b91441a P accfa5b49d7a48e58b1751893eb3ce33: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.005s
I20260812 02:33:45.053893 28023 raft_consensus.cc:348] T d9303f211bc04a449977a6318b91441a P accfa5b49d7a48e58b1751893eb3ce33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "accfa5b49d7a48e58b1751893eb3ce33" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 35951 } } peers { permanent_uuid: "4675861452d9490f9521569aaa091fa8" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 34951 } } peers { permanent_uuid: "e49da65941fe480583c6ea9e169aa8a8" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 43407 } }
I20260812 02:33:45.054246 28023 raft_consensus.cc:374] T d9303f211bc04a449977a6318b91441a P accfa5b49d7a48e58b1751893eb3ce33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:33:45.054306 28023 raft_consensus.cc:729] T d9303f211bc04a449977a6318b91441a P accfa5b49d7a48e58b1751893eb3ce33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: accfa5b49d7a48e58b1751893eb3ce33, State: Initialized, Role: FOLLOWER
I20260812 02:33:45.054845 28023 consensus_queue.cc:260] T d9303f211bc04a449977a6318b91441a P accfa5b49d7a48e58b1751893eb3ce33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "accfa5b49d7a48e58b1751893eb3ce33" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 35951 } } peers { permanent_uuid: "4675861452d9490f9521569aaa091fa8" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 34951 } } peers { permanent_uuid: "e49da65941fe480583c6ea9e169aa8a8" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 43407 } }
I20260812 02:33:45.055665 27996 heartbeater.cc:503] Master 127.14.103.62:41473 was elected leader, sending a full tablet report...
I20260812 02:33:45.056114 28023 ts_tablet_manager.cc:1428] T d9303f211bc04a449977a6318b91441a P accfa5b49d7a48e58b1751893eb3ce33: Time spent starting tablet: real 0.008s user 0.001s sys 0.006s
I20260812 02:33:45.059546 28024 tablet_bootstrap.cc:492] T d9303f211bc04a449977a6318b91441a P 4675861452d9490f9521569aaa091fa8: No bootstrap required, opened a new log
I20260812 02:33:45.059844 28024 ts_tablet_manager.cc:1397] T d9303f211bc04a449977a6318b91441a P 4675861452d9490f9521569aaa091fa8: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.005s
I20260812 02:33:45.064404 28024 raft_consensus.cc:348] T d9303f211bc04a449977a6318b91441a P 4675861452d9490f9521569aaa091fa8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "accfa5b49d7a48e58b1751893eb3ce33" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 35951 } } peers { permanent_uuid: "4675861452d9490f9521569aaa091fa8" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 34951 } } peers { permanent_uuid: "e49da65941fe480583c6ea9e169aa8a8" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 43407 } }
I20260812 02:33:45.064738 28024 raft_consensus.cc:374] T d9303f211bc04a449977a6318b91441a P 4675861452d9490f9521569aaa091fa8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:33:45.064807 28024 raft_consensus.cc:729] T d9303f211bc04a449977a6318b91441a P 4675861452d9490f9521569aaa091fa8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4675861452d9490f9521569aaa091fa8, State: Initialized, Role: FOLLOWER
I20260812 02:33:45.065333 28024 consensus_queue.cc:260] T d9303f211bc04a449977a6318b91441a P 4675861452d9490f9521569aaa091fa8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "accfa5b49d7a48e58b1751893eb3ce33" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 35951 } } peers { permanent_uuid: "4675861452d9490f9521569aaa091fa8" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 34951 } } peers { permanent_uuid: "e49da65941fe480583c6ea9e169aa8a8" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 43407 } }
I20260812 02:33:45.066573 28024 ts_tablet_manager.cc:1428] T d9303f211bc04a449977a6318b91441a P 4675861452d9490f9521569aaa091fa8: Time spent starting tablet: real 0.007s user 0.001s sys 0.005s
W20260812 02:33:45.117964 27839 tablet.cc:2369] T d9303f211bc04a449977a6318b91441a P e49da65941fe480583c6ea9e169aa8a8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 02:33:45.133311 27677 tablet.cc:2369] T d9303f211bc04a449977a6318b91441a P 4675861452d9490f9521569aaa091fa8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 02:33:45.151121 27997 tablet.cc:2369] T d9303f211bc04a449977a6318b91441a P accfa5b49d7a48e58b1751893eb3ce33: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 12 02:33:45 dist-test-slave-2w3w krb5kdc[26875](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502024, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.62@KRBTEST.COM for HTTP/127.14.103.20@KRBTEST.COM
I20260812 02:33:45.207751 28022 raft_consensus.cc:482] T d9303f211bc04a449977a6318b91441a P e49da65941fe480583c6ea9e169aa8a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 02:33:45.208038 28022 raft_consensus.cc:504] T d9303f211bc04a449977a6318b91441a P e49da65941fe480583c6ea9e169aa8a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "accfa5b49d7a48e58b1751893eb3ce33" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 35951 } } peers { permanent_uuid: "4675861452d9490f9521569aaa091fa8" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 34951 } } peers { permanent_uuid: "e49da65941fe480583c6ea9e169aa8a8" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 43407 } }
I20260812 02:33:45.209509 28022 leader_election.cc:288] T d9303f211bc04a449977a6318b91441a P e49da65941fe480583c6ea9e169aa8a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers accfa5b49d7a48e58b1751893eb3ce33 (127.14.103.3:35951), 4675861452d9490f9521569aaa091fa8 (127.14.103.1:34951)
I20260812 02:33:45.215837 28012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:33:45.210171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.2:44459 (local address 127.14.103.3:35951)
0812 02:33:45.210361 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0812 02:33:45.210366 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:33:45.210898 (+ 532us) server_negotiation.cc:407] Connection header received
0812 02:33:45.211157 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:33:45.211161 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:33:45.211215 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:33:45.211264 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:33:45.211727 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:33:45.212504 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:33:45.214284 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:33:45.214934 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:33:45.215097 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:33:45.215170 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0812 02:33:45.215587 (+ 417us) server_negotiation.cc:299] Negotiation successful
0812 02:33:45.215688 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260812 02:33:45.216388 28034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:33:45.209738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.3:35951 (local address 127.14.103.2:44459)
0812 02:33:45.210803 (+ 1065us) negotiation.cc:107] Waiting for socket to connect
0812 02:33:45.210819 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 02:33:45.211021 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:33:45.211423 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:33:45.211426 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:33:45.211457 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:33:45.211599 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:33:45.211605 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:33:45.212681 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:33:45.212685 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:33:45.214113 (+ 1428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:33:45.214126 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:33:45.215288 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:33:45.215291 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:33:45.215428 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:33:45.215454 (+ 26us) client_negotiation.cc:769] Sending connection context
0812 02:33:45.216094 (+ 640us) client_negotiation.cc:240] Negotiation successful
0812 02:33:45.216187 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":993,"thread_start_us":157,"threads_started":1}
I20260812 02:33:45.216471 28015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:33:45.210481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.2:34337 (local address 127.14.103.1:34951)
0812 02:33:45.210631 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0812 02:33:45.210636 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:33:45.211182 (+ 546us) server_negotiation.cc:407] Connection header received
0812 02:33:45.211401 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:33:45.211405 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:33:45.211456 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:33:45.211503 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:33:45.212086 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:33:45.212842 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:33:45.214537 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:33:45.215352 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:33:45.215487 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:33:45.215545 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0812 02:33:45.216238 (+ 693us) server_negotiation.cc:299] Negotiation successful
0812 02:33:45.216335 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260812 02:33:45.217097 27943 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9303f211bc04a449977a6318b91441a" candidate_uuid: "e49da65941fe480583c6ea9e169aa8a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "accfa5b49d7a48e58b1751893eb3ce33" is_pre_election: true
I20260812 02:33:45.217236 28033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:33:45.210300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.1:34951 (local address 127.14.103.2:34337)
0812 02:33:45.211062 (+ 762us) negotiation.cc:107] Waiting for socket to connect
0812 02:33:45.211077 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 02:33:45.211269 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:33:45.211646 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:33:45.211649 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:33:45.211667 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:33:45.211937 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:33:45.211944 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:33:45.213014 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:33:45.213018 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:33:45.214372 (+ 1354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:33:45.214383 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:33:45.215669 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:33:45.215673 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:33:45.215799 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:33:45.215821 (+ 22us) client_negotiation.cc:769] Sending connection context
0812 02:33:45.216991 (+ 1170us) client_negotiation.cc:240] Negotiation successful
0812 02:33:45.217080 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":676,"thread_start_us":190,"threads_started":1}
I20260812 02:33:45.217553 27943 raft_consensus.cc:2454] T d9303f211bc04a449977a6318b91441a P accfa5b49d7a48e58b1751893eb3ce33 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e49da65941fe480583c6ea9e169aa8a8 in term 0.
I20260812 02:33:45.218111 27707 leader_election.cc:302] T d9303f211bc04a449977a6318b91441a P e49da65941fe480583c6ea9e169aa8a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: accfa5b49d7a48e58b1751893eb3ce33, e49da65941fe480583c6ea9e169aa8a8; no voters:
I20260812 02:33:45.218271 27624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9303f211bc04a449977a6318b91441a" candidate_uuid: "e49da65941fe480583c6ea9e169aa8a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4675861452d9490f9521569aaa091fa8" is_pre_election: true
I20260812 02:33:45.218438 28022 raft_consensus.cc:2781] T d9303f211bc04a449977a6318b91441a P e49da65941fe480583c6ea9e169aa8a8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 02:33:45.218510 28022 raft_consensus.cc:482] T d9303f211bc04a449977a6318b91441a P e49da65941fe480583c6ea9e169aa8a8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 02:33:45.218549 28022 raft_consensus.cc:3037] T d9303f211bc04a449977a6318b91441a P e49da65941fe480583c6ea9e169aa8a8 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:33:45.218559 27624 raft_consensus.cc:2454] T d9303f211bc04a449977a6318b91441a P 4675861452d9490f9521569aaa091fa8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e49da65941fe480583c6ea9e169aa8a8 in term 0.
I20260812 02:33:45.219960 28022 raft_consensus.cc:504] T d9303f211bc04a449977a6318b91441a P e49da65941fe480583c6ea9e169aa8a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "accfa5b49d7a48e58b1751893eb3ce33" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 35951 } } peers { permanent_uuid: "4675861452d9490f9521569aaa091fa8" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 34951 } } peers { permanent_uuid: "e49da65941fe480583c6ea9e169aa8a8" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 43407 } }
I20260812 02:33:45.220347 28022 leader_election.cc:288] T d9303f211bc04a449977a6318b91441a P e49da65941fe480583c6ea9e169aa8a8 [CANDIDATE]: Term 1 election: Requested vote from peers accfa5b49d7a48e58b1751893eb3ce33 (127.14.103.3:35951), 4675861452d9490f9521569aaa091fa8 (127.14.103.1:34951)
I20260812 02:33:45.220904 27943 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9303f211bc04a449977a6318b91441a" candidate_uuid: "e49da65941fe480583c6ea9e169aa8a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "accfa5b49d7a48e58b1751893eb3ce33"
I20260812 02:33:45.221081 27943 raft_consensus.cc:3037] T d9303f211bc04a449977a6318b91441a P accfa5b49d7a48e58b1751893eb3ce33 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:33:45.222791 27943 raft_consensus.cc:2454] T d9303f211bc04a449977a6318b91441a P accfa5b49d7a48e58b1751893eb3ce33 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e49da65941fe480583c6ea9e169aa8a8 in term 1.
I20260812 02:33:45.223282 27707 leader_election.cc:302] T d9303f211bc04a449977a6318b91441a P e49da65941fe480583c6ea9e169aa8a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: accfa5b49d7a48e58b1751893eb3ce33, e49da65941fe480583c6ea9e169aa8a8; no voters:
I20260812 02:33:45.223517 28022 raft_consensus.cc:2781] T d9303f211bc04a449977a6318b91441a P e49da65941fe480583c6ea9e169aa8a8 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:33:45.223768 28022 raft_consensus.cc:686] T d9303f211bc04a449977a6318b91441a P e49da65941fe480583c6ea9e169aa8a8 [term 1 LEADER]: Becoming Leader. State: Replica: e49da65941fe480583c6ea9e169aa8a8, State: Running, Role: LEADER
I20260812 02:33:45.224188 28022 consensus_queue.cc:237] T d9303f211bc04a449977a6318b91441a P e49da65941fe480583c6ea9e169aa8a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "accfa5b49d7a48e58b1751893eb3ce33" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 35951 } } peers { permanent_uuid: "4675861452d9490f9521569aaa091fa8" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 34951 } } peers { permanent_uuid: "e49da65941fe480583c6ea9e169aa8a8" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 43407 } }
I20260812 02:33:45.224272 27624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9303f211bc04a449977a6318b91441a" candidate_uuid: "e49da65941fe480583c6ea9e169aa8a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4675861452d9490f9521569aaa091fa8"
I20260812 02:33:45.224455 27624 raft_consensus.cc:3037] T d9303f211bc04a449977a6318b91441a P 4675861452d9490f9521569aaa091fa8 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:33:45.225926 27624 raft_consensus.cc:2454] T d9303f211bc04a449977a6318b91441a P 4675861452d9490f9521569aaa091fa8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e49da65941fe480583c6ea9e169aa8a8 in term 1.
I20260812 02:33:45.228006 27428 catalog_manager.cc:5495] T d9303f211bc04a449977a6318b91441a P e49da65941fe480583c6ea9e169aa8a8 reported cstate change: term changed from 0 to 1, leader changed from <none> to e49da65941fe480583c6ea9e169aa8a8 (127.14.103.2). New cstate: current_term: 1 leader_uuid: "e49da65941fe480583c6ea9e169aa8a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "accfa5b49d7a48e58b1751893eb3ce33" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 35951 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4675861452d9490f9521569aaa091fa8" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 34951 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e49da65941fe480583c6ea9e169aa8a8" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 43407 } health_report { overall_health: HEALTHY } } }
I20260812 02:33:45.305728 28018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:33:45.300098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37734 (local address 127.14.103.2:43407)
0812 02:33:45.300257 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0812 02:33:45.300263 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 02:33:45.300281 (+ 18us) server_negotiation.cc:407] Connection header received
0812 02:33:45.300399 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:33:45.300403 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:33:45.300481 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:33:45.300609 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 02:33:45.301027 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:33:45.301822 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:33:45.303189 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:33:45.303541 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:33:45.303655 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:33:45.304143 (+ 488us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 02:33:45.304801 (+ 658us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 02:33:45.304933 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0812 02:33:45.305492 (+ 559us) server_negotiation.cc:299] Negotiation successful
0812 02:33:45.305571 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260812 02:33:45.314690 27624 raft_consensus.cc:1261] T d9303f211bc04a449977a6318b91441a P 4675861452d9490f9521569aaa091fa8 [term 1 FOLLOWER]: Refusing update from remote peer e49da65941fe480583c6ea9e169aa8a8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:33:45.314924 27943 raft_consensus.cc:1261] T d9303f211bc04a449977a6318b91441a P accfa5b49d7a48e58b1751893eb3ce33 [term 1 FOLLOWER]: Refusing update from remote peer e49da65941fe480583c6ea9e169aa8a8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:33:45.315779 28035 consensus_queue.cc:1035] T d9303f211bc04a449977a6318b91441a P e49da65941fe480583c6ea9e169aa8a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4675861452d9490f9521569aaa091fa8" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 34951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:33:45.316432 28035 consensus_queue.cc:1035] T d9303f211bc04a449977a6318b91441a P e49da65941fe480583c6ea9e169aa8a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "accfa5b49d7a48e58b1751893eb3ce33" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 35951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:33:45.345952 28038 mvcc.cc:205] Tried to move back new op lower bound from 7317512295678308352 to 7317512295321698304. Current Snapshot: MvccSnapshot[applied={T|T < 7317512295678308352}]
I20260812 02:33:45.348464 28041 mvcc.cc:205] Tried to move back new op lower bound from 7317512295678308352 to 7317512295321698304. Current Snapshot: MvccSnapshot[applied={T|T < 7317512295678308352}]
I20260812 02:33:45.355072 28039 mvcc.cc:205] Tried to move back new op lower bound from 7317512295678308352 to 7317512295321698304. Current Snapshot: MvccSnapshot[applied={T|T < 7317512295678308352}]
I20260812 02:33:45.367838 27433 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48932:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 02:33:45.368060 27433 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48932:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 02:33:45.371208 27433 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5801c57365684accaafb51587361b050: Sending DeleteTablet for 3 replicas of tablet d9303f211bc04a449977a6318b91441a
I20260812 02:33:45.372455 27579 tablet_service.cc:1514] Processing DeleteTablet for tablet d9303f211bc04a449977a6318b91441a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:33:45 UTC) from {username='kudu', principal='kudu/127.14.103.62@KRBTEST.COM'} at 127.0.0.1:45906
I20260812 02:33:45.373677 27917 tablet_service.cc:1514] Processing DeleteTablet for tablet d9303f211bc04a449977a6318b91441a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:33:45 UTC) from {username='kudu', principal='kudu/127.14.103.62@KRBTEST.COM'} at 127.0.0.1:60422
I20260812 02:33:45.374501 27757 tablet_service.cc:1514] Processing DeleteTablet for tablet d9303f211bc04a449977a6318b91441a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:33:45 UTC) from {username='kudu', principal='kudu/127.14.103.62@KRBTEST.COM'} at 127.0.0.1:37726
I20260812 02:33:45.376061 14748 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbep4M4/build/debug/bin/kudu with pid 27524
I20260812 02:33:45.378875 28048 tablet_replica.cc:330] T d9303f211bc04a449977a6318b91441a P accfa5b49d7a48e58b1751893eb3ce33: stopping tablet replica
I20260812 02:33:45.379366 28048 raft_consensus.cc:2229] T d9303f211bc04a449977a6318b91441a P accfa5b49d7a48e58b1751893eb3ce33 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 02:33:45.379753 28048 raft_consensus.cc:2258] T d9303f211bc04a449977a6318b91441a P accfa5b49d7a48e58b1751893eb3ce33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:33:45.383349 28049 tablet_replica.cc:330] T d9303f211bc04a449977a6318b91441a P e49da65941fe480583c6ea9e169aa8a8: stopping tablet replica
I20260812 02:33:45.384558 28048 ts_tablet_manager.cc:1905] T d9303f211bc04a449977a6318b91441a P accfa5b49d7a48e58b1751893eb3ce33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:33:45.386349 28049 raft_consensus.cc:2229] T d9303f211bc04a449977a6318b91441a P e49da65941fe480583c6ea9e169aa8a8 [term 1 LEADER]: Raft consensus shutting down.
I20260812 02:33:45.386927 28049 raft_consensus.cc:2258] T d9303f211bc04a449977a6318b91441a P e49da65941fe480583c6ea9e169aa8a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:33:45.389398 28049 ts_tablet_manager.cc:1905] T d9303f211bc04a449977a6318b91441a P e49da65941fe480583c6ea9e169aa8a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:33:45.392694 28049 ts_tablet_manager.cc:1918] T d9303f211bc04a449977a6318b91441a P e49da65941fe480583c6ea9e169aa8a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 02:33:45.392833 28049 log.cc:1196] T d9303f211bc04a449977a6318b91441a P e49da65941fe480583c6ea9e169aa8a8: Deleting WAL directory at /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501898713699-14748-0/minicluster-data/ts-1/wal/wals/d9303f211bc04a449977a6318b91441a
I20260812 02:33:45.393203 28049 ts_tablet_manager.cc:1939] T d9303f211bc04a449977a6318b91441a P e49da65941fe480583c6ea9e169aa8a8: Deleting consensus metadata
W20260812 02:33:45.393946 27415 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.103.1:34951 (error 108)
I20260812 02:33:45.387924 28048 ts_tablet_manager.cc:1918] T d9303f211bc04a449977a6318b91441a P accfa5b49d7a48e58b1751893eb3ce33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 02:33:45.394793 14748 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbep4M4/build/debug/bin/kudu with pid 27687
I20260812 02:33:45.399493 28048 log.cc:1196] T d9303f211bc04a449977a6318b91441a P accfa5b49d7a48e58b1751893eb3ce33: Deleting WAL directory at /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786501898713699-14748-0/minicluster-data/ts-2/wal/wals/d9303f211bc04a449977a6318b91441a
I20260812 02:33:45.400115 28048 ts_tablet_manager.cc:1939] T d9303f211bc04a449977a6318b91441a P accfa5b49d7a48e58b1751893eb3ce33: Deleting consensus metadata
I20260812 02:33:45.402323 27413 catalog_manager.cc:4841] TS accfa5b49d7a48e58b1751893eb3ce33 (127.14.103.3:35951): tablet d9303f211bc04a449977a6318b91441a (table test-table [id=b0c95542827e4bd39f48e98fdd0df74c]) successfully deleted
I20260812 02:33:45.403911 28011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:33:45.403391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.1:34951 (local address 127.0.0.1:45910)
0812 02:33:45.403667 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0812 02:33:45.403784 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.1:34951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159}
W20260812 02:33:45.404157 27415 catalog_manager.cc:4584] TS 4675861452d9490f9521569aaa091fa8 (127.14.103.1:34951): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d9303f211bc04a449977a6318b91441a: Network error: Client connection negotiation failed: client connection to 127.14.103.1:34951: connect: Connection refused (error 111)
W20260812 02:33:45.416934 28011 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:33:45.416298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.2:43407 (local address 127.0.0.1:37746)
0812 02:33:45.416500 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0812 02:33:45.416516 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 02:33:45.416612 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:33:45.416828 (+ 216us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.2:43407: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":122}
I20260812 02:33:45.417672 14748 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbep4M4/build/debug/bin/kudu with pid 27843
I20260812 02:33:45.426106 14748 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbep4M4/build/debug/bin/kudu with pid 27394
I20260812 02:33:45.436712 14748 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260812 02:33:45.783285 14748 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260812 02:33:45.783459 14748 mini_ranger.cc:66] Stopping Ranger...
I20260812 02:33:46.131788 14748 mini_ranger.cc:68] Stopped Ranger
2026-08-12 02:33:46.132 UTC [26911] LOG: received smart shutdown request
2026-08-12 02:33:46.137 UTC [26911] LOG: background worker "logical replication launcher" (PID 26925) exited with exit code 1
2026-08-12 02:33:46.138 UTC [26919] LOG: shutting down
2026-08-12 02:33:46.385 UTC [26911] LOG: database system is shut down
2026-08-12T02:33:46Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (55483 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 02:33:46 dist-test-slave-2w3w krb5kdc[28062](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 02:33:46 dist-test-slave-2w3w krb5kdc[28062](info): set up 2 sockets
Aug 12 02:33:46 dist-test-slave-2w3w krb5kdc[28062](info): commencing operation
krb5kdc: starting...
W20260812 02:33:47.717748 14748 mini_kdc.cc:121] Time spent starting KDC: real 1.164s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/krb5kdc/test-user.keytab.
Aug 12 02:33:47 dist-test-slave-2w3w krb5kdc[28062](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502027, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T02:33:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T02:33:47Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.14.103.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.14.103.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.14.103.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/krb5kdc/rangeradmin_127.14.103.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.14.103.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/krb5kdc/rangeradmin_127.14.103.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.14.103.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.14.103.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.14.103.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/krb5kdc/rangerlookup_127.14.103.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.14.103.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/krb5kdc/rangerlookup_127.14.103.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.14.103.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.14.103.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.103.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/krb5kdc/HTTP_127.14.103.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.14.103.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/krb5kdc/HTTP_127.14.103.20@KRBTEST.COM.keytab.
I20260812 02:33:47.939905 14748 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskbep4M4/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/minicluster-data/postgres -l logfile start
2026-08-12 02:33:51.353 UTC [28098] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-12 02:33:51.353 UTC [28098] LOG: listening on IPv4 address "127.14.103.20", port 37145
2026-08-12 02:33:51.360 UTC [28098] LOG: listening on Unix socket "/tmp/.s.PGSQL.37145"
2026-08-12 02:33:51.388 UTC [28102] LOG: database system was shut down at 2026-08-12 02:33:49 UTC
2026-08-12 02:33:51.393 UTC [28098] LOG: database system is ready to accept connections
I20260812 02:33:52.492452 14748 mini_postgres.cc:96] Postgres bound to 37145
2026-08-12 02:33:52.501 UTC [28110] FATAL: database "slave" does not exist
127.14.103.20:37145 - accepting connections
I20260812 02:33:52.502244 14748 mini_ranger.cc:161] Starting Ranger...
I20260812 02:33:52.522403 14748 mini_ranger.cc:84] Created miniranger Postgres user
I20260812 02:33:53.318697 14748 mini_ranger.cc:87] Created ranger Postgres database
I20260812 02:33:53.318801 14748 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/minicluster-data/ranger-admin
2026-08-12 02:33:53,659 [I] DB FLAVOR :POSTGRES
2026-08-12 02:33:53,659 [I] --------- Verifying Ranger DB connection ---------
2026-08-12 02:33:53,659 [I] Checking connection..
2026-08-12 02:33:53,659 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbep4M4/build/debug/bin/postgresql.jar:/tmp/dist-test-taskbep4M4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.103.20:37145/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-12 02:33:53,992 [I] Checking connection passed.
2026-08-12 02:33:53,992 [I] --------- Verifying version history table ---------
2026-08-12 02:33:53,993 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbep4M4/build/debug/bin/postgresql.jar:/tmp/dist-test-taskbep4M4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.103.20:37145/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-12 02:33:54,361 [I] Table x_db_version_h does not exist in database ranger
2026-08-12 02:33:54,362 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbep4M4/build/debug/bin/postgresql.jar:/tmp/dist-test-taskbep4M4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.103.20:37145/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-12 02:33:54,683 [I] Table x_db_version_h does not exist in database ranger
2026-08-12 02:33:54,683 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-12 02:33:54,683 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbep4M4/build/debug/bin/postgresql.jar:/tmp/dist-test-taskbep4M4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.103.20:37145/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskbep4M4/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-12 02:33:55.021 UTC [28182] WARNING: there is no transaction in progress
2026-08-12 02:33:55,036 [I] create_dbversion_catalog.sql file imported successfully
2026-08-12 02:33:55,037 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbep4M4/build/debug/bin/postgresql.jar:/tmp/dist-test-taskbep4M4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.103.20:37145/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-12 02:33:55,431 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-12 02:33:55,431 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-12 02:33:55,432 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbep4M4/build/debug/bin/postgresql.jar:/tmp/dist-test-taskbep4M4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.103.20:37145/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-12 02:33:55,782 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbep4M4/build/debug/bin/postgresql.jar:/tmp/dist-test-taskbep4M4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.103.20:37145/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-12 02:33:56,111 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbep4M4/build/debug/bin/postgresql.jar:/tmp/dist-test-taskbep4M4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.103.20:37145/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-2w3w.c.gcp-upstream.internal','N') ;"
2026-08-12 02:33:56,411 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbep4M4/build/debug/bin/postgresql.jar:/tmp/dist-test-taskbep4M4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.103.20:37145/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-12 02:33:56,734 [I] Table x_portal_user does not exist in database ranger
2026-08-12 02:33:56,734 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbep4M4/build/debug/bin/postgresql.jar:/tmp/dist-test-taskbep4M4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.103.20:37145/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-12 02:33:57,044 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-12 02:33:57,044 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbep4M4/build/debug/bin/postgresql.jar:/tmp/dist-test-taskbep4M4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.103.20:37145/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-12 02:33:57,331 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbep4M4/build/debug/bin/postgresql.jar:/tmp/dist-test-taskbep4M4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.103.20:37145/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-12 02:33:57,601 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-12 02:33:57,602 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbep4M4/build/debug/bin/postgresql.jar:/tmp/dist-test-taskbep4M4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.103.20:37145/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskbep4M4/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-12 02:33:58.945 UTC [28304] WARNING: there is no transaction in progress
2026-08-12 02:33:58.968 UTC [28304] WARNING: there is no transaction in progress
2026-08-12 02:33:58.991 UTC [28304] WARNING: there is no transaction in progress
2026-08-12 02:33:59.018 UTC [28304] WARNING: there is no transaction in progress
2026-08-12 02:33:59.040 UTC [28304] WARNING: there is no transaction in progress
2026-08-12 02:33:59.067 UTC [28304] WARNING: there is no transaction in progress
2026-08-12 02:33:59.153 UTC [28304] WARNING: there is no transaction in progress
2026-08-12 02:33:59.163 UTC [28304] WARNING: there is no transaction in progress
2026-08-12 02:33:59.182 UTC [28304] WARNING: there is no transaction in progress
2026-08-12 02:33:59.197 UTC [28304] WARNING: there is no transaction in progress
2026-08-12 02:33:59.211 UTC [28304] WARNING: there is no transaction in progress
2026-08-12 02:33:59.229 UTC [28304] WARNING: there is no transaction in progress
2026-08-12 02:33:59.243 UTC [28304] WARNING: there is no transaction in progress
2026-08-12 02:33:59.252 UTC [28304] WARNING: there is no transaction in progress
2026-08-12 02:33:59.263 UTC [28304] WARNING: there is no transaction in progress
2026-08-12 02:33:59.274 UTC [28304] WARNING: there is no transaction in progress
2026-08-12 02:34:00,236 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-12 02:34:00,237 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbep4M4/build/debug/bin/postgresql.jar:/tmp/dist-test-taskbep4M4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.103.20:37145/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-12 02:34:00,506 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-12 02:34:00,507 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbep4M4/build/debug/bin/postgresql.jar:/tmp/dist-test-taskbep4M4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.103.20:37145/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-12 02:34:00,791 [I] Table x_portal_user already exists in database 'ranger'
2026-08-12 02:34:00,792 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbep4M4/build/debug/bin/postgresql.jar:/tmp/dist-test-taskbep4M4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.103.20:37145/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-12 02:34:01,099 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-12 02:34:01,099 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbep4M4/build/debug/bin/postgresql.jar:/tmp/dist-test-taskbep4M4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.103.20:37145/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-12 02:34:01,378 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbep4M4/build/debug/bin/postgresql.jar:/tmp/dist-test-taskbep4M4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.103.20:37145/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-12 02:34:01,661 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbep4M4/build/debug/bin/postgresql.jar:/tmp/dist-test-taskbep4M4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.103.20:37145/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-2w3w.c.gcp-upstream.internal';"
2026-08-12 02:34:01,930 [I] CORE_DB_SCHEMA import status has been updated
2026-08-12 02:34:01,930 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbep4M4/build/debug/bin/postgresql.jar:/tmp/dist-test-taskbep4M4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.103.20:37145/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-12 02:34:02,239 [I] DB_PATCHES have already been applied
I20260812 02:34:02.245311 14748 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/minicluster-data/ranger-admin:/tmp/dist-test-taskbep4M4/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskbep4M4/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskbep4M4/build/debug/bin/postgresql.jar:/tmp/dist-test-taskbep4M4/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260812 02:34:02.245396 14748 mini_ranger.cc:193] Using host: 127.14.103.20
I20260812 02:34:02.250671 14748 mini_ranger.cc:227] Ranger admin URL: http://127.14.103.20:54775
Aug 12, 2026 2:34:02 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 12, 2026 2:34:02 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 12, 2026 2:34:02 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/minicluster-data/ranger-admin/ews/webapp
Aug 12, 2026 2:34:02 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 12, 2026 2:34:02 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 12, 2026 2:34:03 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/minicluster-data/ranger-admin/ews/webapp].
Aug 12, 2026 2:34:03 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260812 02:34:03.510936 14748 mini_ranger.cc:160] Time spent starting Ranger: real 11.009s user 0.001s sys 0.014s
Aug 12, 2026 2:34:27 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 12, 2026 2:34:27 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Aug 12, 2026 2:34:27 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 12, 2026 2:34:27 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 12, 2026 2:34:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 12, 2026 2:34:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 12, 2026 2:34:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 12, 2026 2:34:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 12, 2026 2:34:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 12, 2026 2:34:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 12, 2026 2:34:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 12, 2026 2:34:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 12, 2026 2:34:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 12, 2026 2:34:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 12, 2026 2:34:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 12, 2026 2:34:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 12, 2026 2:34:27 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 12, 2026 2:34:28 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260812 02:34:29.455765 14748 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.14.103.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.14.103.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.14.103.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/krb5kdc/rangerkms_127.14.103.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.14.103.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/krb5kdc/rangerkms_127.14.103.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.14.103.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.14.103.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.103.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/krb5kdc/HTTP_127.14.103.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.14.103.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/krb5kdc/HTTP_127.14.103.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 12 02:34:29 dist-test-slave-2w3w krb5kdc[28062](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502069, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260812 02:34:29.614102 14748 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260812 02:34:29.635695 14748 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260812 02:34:30.979787 14748 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260812 02:34:30.979934 14748 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/minicluster-data/ranger-kms
I20260812 02:34:30.979959 14748 mini_ranger_kms.cc:227] Using postgres at 127.14.103.20:37145
2026-08-12 02:34:31,200 [I] DB FLAVOR :POSTGRES
2026-08-12 02:34:31,200 [I] --------- Verifying Ranger DB connection ---------
2026-08-12 02:34:31,200 [I] Checking connection
2026-08-12 02:34:31,200 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbep4M4/build/debug/bin/postgresql.jar:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.103.20:37145/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-12 02:34:31,471 [I] connection success
2026-08-12 02:34:31,471 [I] --------- Verifying Ranger DB tables ---------
2026-08-12 02:34:31,472 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-12 02:34:31,472 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbep4M4/build/debug/bin/postgresql.jar:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.103.20:37145/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-12 02:34:31,753 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-12 02:34:31,754 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-12 02:34:31,754 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-12 02:34:31,754 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbep4M4/build/debug/bin/postgresql.jar:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.103.20:37145/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-12 02:34:32,107 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260812 02:34:32.810164 14748 mini_ranger_kms.cc:310] Created kms service
I20260812 02:34:33.026924 14748 mini_ranger_kms.cc:326] Created kudu user
I20260812 02:34:33.094931 14748 mini_ranger_kms.cc:343] Created rangerkms user
Aug 12 02:34:33 dist-test-slave-2w3w krb5kdc[28062](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502069, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.14.103.20@KRBTEST.COM
I20260812 02:34:33.592412 14748 mini_ranger_kms.cc:384] Added ranger policy
I20260812 02:34:33.592622 14748 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/minicluster-data/ranger-kms:/tmp/dist-test-taskbep4M4/build/debug/bin/postgresql.jar:/tmp/dist-test-taskbep4M4/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskbep4M4/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260812 02:34:33.592685 14748 mini_ranger_kms.cc:240] Using host: 127.14.103.20
I20260812 02:34:33.598567 14748 mini_ranger_kms.cc:276] Ranger KMS PID: 28513
I20260812 02:34:33.598678 14748 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.14.103.20:59163
Aug 12, 2026 2:34:34 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 12, 2026 2:34:34 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 12, 2026 2:34:34 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 12, 2026 2:34:34 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 12, 2026 2:34:35 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/minicluster-data/ranger-kms/ews/webapp].
Aug 12, 2026 2:34:35 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 12 02:34:35 dist-test-slave-2w3w krb5kdc[28062](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786502075, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.14.103.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:34:36.068842 14748 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 6.455s user 0.011s sys 0.195s
I20260812 02:34:36.069011 14748 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260812 02:34:36.069123 14748 mini_ranger_kms.cc:401] 127.14.103.20:59163/kms/v1/keys
Aug 12 02:34:36 dist-test-slave-2w3w krb5kdc[28062](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502075, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.14.103.20@KRBTEST.COM for HTTP/127.14.103.20@KRBTEST.COM
Aug 12 02:34:36 dist-test-slave-2w3w krb5kdc[28062](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502075, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.14.103.20@KRBTEST.COM for HTTP/127.14.103.20@KRBTEST.COM
Aug 12 02:34:39 dist-test-slave-2w3w krb5kdc[28062](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502079, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.14.103.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.103.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.103.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.62.keytab.
Entry for principal kudu/127.14.103.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.62.keytab.
I20260812 02:34:39.335129 14748 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbep4M4/build/debug/bin/kudu
/tmp/dist-test-taskbep4M4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.103.62:35127
--webserver_interface=127.14.103.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.62.keytab
--principal=kudu/127.14.103.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.103.20:45441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.103.62:35127
--ranger_config_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.103.20:59163/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:34:39.474401 28571 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:34:39.474807 28571 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:34:39.474905 28571 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:34:39.479296 28571 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:34:39.479432 28571 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:34:39.479470 28571 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:34:39.479512 28571 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:34:39.479549 28571 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 02:34:39.485831 28571 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.103.20:45441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/minicluster-data/master-0/wal
--ranger_kms_url=127.14.103.20:59163/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.14.103.62:35127
--ranger_config_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.62.keytab
--principal=kudu/127.14.103.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.103.62:35127
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.103.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28571
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:34:39.487462 28571 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:34:39.488785 28571 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:34:39.499727 28583 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:34:39.499737 28579 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:34:39.499899 28571 server_base.cc:1034] running on GCE node
W20260812 02:34:39.499954 28578 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:34:39.501336 28571 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:34:39.502662 28571 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:34:39.503914 28571 hybrid_clock.cc:648] HybridClock initialized: now 1786502079503904 us; error 54 us; skew 500 ppm
Aug 12 02:34:39 dist-test-slave-2w3w krb5kdc[28062](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502079, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:34:39.508119 28571 init.cc:377] Logged in from keytab as kudu/127.14.103.62@KRBTEST.COM (short username kudu)
I20260812 02:34:39.509303 28571 webserver.cc:460] Webserver started at http://127.14.103.62:36527/ using document root <none> and password file <none>
I20260812 02:34:39.509934 28571 fs_manager.cc:359] Metadata directory not provided
I20260812 02:34:39.510033 28571 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:34:39.510391 28571 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 12 02:34:39 dist-test-slave-2w3w krb5kdc[28062](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502079, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.62@KRBTEST.COM for HTTP/127.14.103.20@KRBTEST.COM
I20260812 02:34:39.565672 28571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/minicluster-data/master-0/data/instance:
uuid: "2b19b08f7ef849d698d640245912b1e4"
format_stamp: "Formatted at 2026-08-12 02:34:39 on dist-test-slave-2w3w"
server_key: "e4bc8128b5e5fc4ee11cae35cbd6a886"
server_key_iv: "0e98e13f00b9990507e8eca7de64f85a"
server_key_version: "kuduclusterkey@0"
I20260812 02:34:39.566484 28571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/minicluster-data/master-0/wal/instance:
uuid: "2b19b08f7ef849d698d640245912b1e4"
format_stamp: "Formatted at 2026-08-12 02:34:39 on dist-test-slave-2w3w"
server_key: "e4bc8128b5e5fc4ee11cae35cbd6a886"
server_key_iv: "0e98e13f00b9990507e8eca7de64f85a"
server_key_version: "kuduclusterkey@0"
I20260812 02:34:39.571326 28571 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260812 02:34:39.593356 28591 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:34:39.595628 28571 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.005s sys 0.000s
I20260812 02:34:39.595846 28571 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/minicluster-data/master-0/data,/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/minicluster-data/master-0/wal
uuid: "2b19b08f7ef849d698d640245912b1e4"
format_stamp: "Formatted at 2026-08-12 02:34:39 on dist-test-slave-2w3w"
server_key: "e4bc8128b5e5fc4ee11cae35cbd6a886"
server_key_iv: "0e98e13f00b9990507e8eca7de64f85a"
server_key_version: "kuduclusterkey@0"
I20260812 02:34:39.596012 28571 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:34:39.617668 28571 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:34:39.622432 28571 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:34:39.622709 28571 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:34:39.632036 28571 rpc_server.cc:307] RPC server started. Bound to: 127.14.103.62:35127
I20260812 02:34:39.632042 28651 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.103.62:35127 every 8 connection(s)
I20260812 02:34:39.633258 28571 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/minicluster-data/master-0/data/info.pb
I20260812 02:34:39.635890 14748 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbep4M4/build/debug/bin/kudu as pid 28571
I20260812 02:34:39.636032 14748 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/minicluster-data/master-0/wal/instance
I20260812 02:34:39.636355 14748 external_mini_cluster.cc:1428] Setting key ce96ab029fcfd664cb36841fe1fc82ac
I20260812 02:34:39.636821 28652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:34:39.645010 28652 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b19b08f7ef849d698d640245912b1e4: Bootstrap starting.
Aug 12 02:34:39 dist-test-slave-2w3w krb5kdc[28062](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502079, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.103.62@KRBTEST.COM
I20260812 02:34:39.647921 28652 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2b19b08f7ef849d698d640245912b1e4: Neither blocks nor log segments found. Creating new log.
I20260812 02:34:39.648940 28652 log.cc:824] T 00000000000000000000000000000000 P 2b19b08f7ef849d698d640245912b1e4: Log is configured to *not* fsync() on all Append() calls
I20260812 02:34:39.652076 28652 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b19b08f7ef849d698d640245912b1e4: No bootstrap required, opened a new log
I20260812 02:34:39.656068 28655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:39.638005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57726 (local address 127.14.103.62:35127)
0812 02:34:39.638512 (+ 507us) server_negotiation.cc:206] Beginning negotiation
0812 02:34:39.638536 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0812 02:34:39.638583 (+ 47us) server_negotiation.cc:407] Connection header received
0812 02:34:39.639502 (+ 919us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:34:39.639528 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:34:39.640007 (+ 479us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:34:39.640446 (+ 439us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:34:39.641482 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:39.643055 (+ 1573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:39.644113 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:39.644857 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:39.645057 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:39.648642 (+ 3585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:34:39.648653 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:34:39.648678 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:34:39.648724 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:34:39.651732 (+ 3008us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:34:39.652408 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:34:39.652417 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:34:39.652424 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:34:39.652588 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:34:39.653284 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:34:39.653288 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:34:39.653290 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:34:39.653885 (+ 595us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:34:39.654232 (+ 347us) server_negotiation.cc:1035] Waiting for connection context
0812 02:34:39.654601 (+ 369us) server_negotiation.cc:299] Negotiation successful
0812 02:34:39.654939 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":150,"threads_started":1}
I20260812 02:34:39.656514 28652 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2b19b08f7ef849d698d640245912b1e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b19b08f7ef849d698d640245912b1e4" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 35127 } }
I20260812 02:34:39.656814 28652 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2b19b08f7ef849d698d640245912b1e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:34:39.656867 28652 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2b19b08f7ef849d698d640245912b1e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b19b08f7ef849d698d640245912b1e4, State: Initialized, Role: FOLLOWER
I20260812 02:34:39.657393 28652 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2b19b08f7ef849d698d640245912b1e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b19b08f7ef849d698d640245912b1e4" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 35127 } }
I20260812 02:34:39.657574 28652 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2b19b08f7ef849d698d640245912b1e4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 02:34:39.657666 28652 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2b19b08f7ef849d698d640245912b1e4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 02:34:39.657790 28652 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2b19b08f7ef849d698d640245912b1e4 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:34:39.659148 28652 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2b19b08f7ef849d698d640245912b1e4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b19b08f7ef849d698d640245912b1e4" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 35127 } }
I20260812 02:34:39.659533 28652 leader_election.cc:302] T 00000000000000000000000000000000 P 2b19b08f7ef849d698d640245912b1e4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2b19b08f7ef849d698d640245912b1e4; no voters:
I20260812 02:34:39.659879 28652 leader_election.cc:288] T 00000000000000000000000000000000 P 2b19b08f7ef849d698d640245912b1e4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 02:34:39.660019 28657 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2b19b08f7ef849d698d640245912b1e4 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:34:39.660387 28657 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2b19b08f7ef849d698d640245912b1e4 [term 1 LEADER]: Becoming Leader. State: Replica: 2b19b08f7ef849d698d640245912b1e4, State: Running, Role: LEADER
I20260812 02:34:39.660774 28657 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2b19b08f7ef849d698d640245912b1e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b19b08f7ef849d698d640245912b1e4" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 35127 } }
I20260812 02:34:39.661340 28652 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2b19b08f7ef849d698d640245912b1e4 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:34:39.662732 28660 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b19b08f7ef849d698d640245912b1e4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2b19b08f7ef849d698d640245912b1e4. Latest consensus state: current_term: 1 leader_uuid: "2b19b08f7ef849d698d640245912b1e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b19b08f7ef849d698d640245912b1e4" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 35127 } } }
I20260812 02:34:39.663079 28660 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b19b08f7ef849d698d640245912b1e4 [sys.catalog]: This master's current role is: LEADER
I20260812 02:34:39.662776 28659 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b19b08f7ef849d698d640245912b1e4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2b19b08f7ef849d698d640245912b1e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b19b08f7ef849d698d640245912b1e4" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 35127 } } }
I20260812 02:34:39.663254 28659 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b19b08f7ef849d698d640245912b1e4 [sys.catalog]: This master's current role is: LEADER
I20260812 02:34:39.664083 28669 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 02:34:39.664497 28652 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260812 02:34:39.670713 28669 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 02:34:39.682107 28669 catalog_manager.cc:1347] Generated new cluster ID: a6819f31033d4d55a8c8065b815ee143
I20260812 02:34:39.682292 28669 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 02:34:39.716041 28669 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 02:34:39.717516 28669 catalog_manager.cc:1504] Loading token signing keys...
I20260812 02:34:39.730417 28669 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2b19b08f7ef849d698d640245912b1e4: Generated new TSK 0
I20260812 02:34:39.731487 28669 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.103.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.103.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.103.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.1.keytab.
Entry for principal kudu/127.14.103.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.1.keytab.
I20260812 02:34:40.499931 14748 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbep4M4/build/debug/bin/kudu
/tmp/dist-test-taskbep4M4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.103.20:59163/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.103.1:0
--local_ip_for_outbound_sockets=127.14.103.1
--webserver_interface=127.14.103.1
--webserver_port=0
--tserver_master_addrs=127.14.103.62:35127
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.1.keytab
--principal=kudu/127.14.103.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.103.20:45441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:34:40.650933 28701 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:34:40.651356 28701 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:34:40.651461 28701 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:34:40.656121 28701 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:34:40.656231 28701 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:34:40.656360 28701 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.103.1
I20260812 02:34:40.662938 28701 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.103.20:45441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.14.103.20:59163/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.1.keytab
--principal=kudu/127.14.103.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.103.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.103.1
--webserver_port=0
--tserver_master_addrs=127.14.103.62:35127
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28701
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.103.1
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:34:40.664290 28701 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:34:40.665405 28701 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:34:40.678009 28712 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:34:40.678009 28708 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:34:40.678337 28709 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:34:40.679805 28701 server_base.cc:1034] running on GCE node
I20260812 02:34:40.680354 28701 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:34:40.681133 28701 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:34:40.682361 28701 hybrid_clock.cc:648] HybridClock initialized: now 1786502080682323 us; error 52 us; skew 500 ppm
Aug 12 02:34:40 dist-test-slave-2w3w krb5kdc[28062](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502080, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:34:40.685910 28701 init.cc:377] Logged in from keytab as kudu/127.14.103.1@KRBTEST.COM (short username kudu)
I20260812 02:34:40.687165 28701 webserver.cc:460] Webserver started at http://127.14.103.1:33037/ using document root <none> and password file <none>
I20260812 02:34:40.687794 28701 fs_manager.cc:359] Metadata directory not provided
I20260812 02:34:40.687897 28701 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:34:40.688160 28701 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 12 02:34:40 dist-test-slave-2w3w krb5kdc[28062](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502080, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.1@KRBTEST.COM for HTTP/127.14.103.20@KRBTEST.COM
I20260812 02:34:40.706008 28701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/minicluster-data/ts-0/data/instance:
uuid: "3a6a0ec8e201413f8e57ba328d3e2f73"
format_stamp: "Formatted at 2026-08-12 02:34:40 on dist-test-slave-2w3w"
server_key: "5c3d31e9c422523620f4edddfa66a22c"
server_key_iv: "c1720fc66ec2c24989828037627c3234"
server_key_version: "kuduclusterkey@0"
I20260812 02:34:40.707053 28701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/minicluster-data/ts-0/wal/instance:
uuid: "3a6a0ec8e201413f8e57ba328d3e2f73"
format_stamp: "Formatted at 2026-08-12 02:34:40 on dist-test-slave-2w3w"
server_key: "5c3d31e9c422523620f4edddfa66a22c"
server_key_iv: "c1720fc66ec2c24989828037627c3234"
server_key_version: "kuduclusterkey@0"
I20260812 02:34:40.712807 28701 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260812 02:34:40.747467 28721 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:34:40.749512 28701 fs_manager.cc:725] Time spent opening block manager: real 0.020s user 0.003s sys 0.000s
I20260812 02:34:40.749683 28701 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/minicluster-data/ts-0/wal
uuid: "3a6a0ec8e201413f8e57ba328d3e2f73"
format_stamp: "Formatted at 2026-08-12 02:34:40 on dist-test-slave-2w3w"
server_key: "5c3d31e9c422523620f4edddfa66a22c"
server_key_iv: "c1720fc66ec2c24989828037627c3234"
server_key_version: "kuduclusterkey@0"
I20260812 02:34:40.749837 28701 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:34:40.767284 28701 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:34:40.771291 28701 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:34:40.771581 28701 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:34:40.772415 28701 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:34:40.773591 28701 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:34:40.773700 28701 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:34:40.773830 28701 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:34:40.773928 28701 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:34:40.790462 28701 rpc_server.cc:307] RPC server started. Bound to: 127.14.103.1:42415
I20260812 02:34:40.791982 28701 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/minicluster-data/ts-0/data/info.pb
I20260812 02:34:40.794358 28851 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.103.1:42415 every 8 connection(s)
I20260812 02:34:40.801064 14748 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbep4M4/build/debug/bin/kudu as pid 28701
I20260812 02:34:40.801215 14748 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/minicluster-data/ts-0/wal/instance
I20260812 02:34:40.801558 14748 external_mini_cluster.cc:1428] Setting key 76171bc3ee08781c0adec7f7d04c8806
Aug 12 02:34:40 dist-test-slave-2w3w krb5kdc[28062](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502080, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.1@KRBTEST.COM for kudu/127.14.103.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.14.103.2@KRBTEST.COM; defaulting to no policy
I20260812 02:34:40.815455 28854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:40.796455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.1:37489 (local address 127.14.103.62:35127)
0812 02:34:40.796761 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0812 02:34:40.796766 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:34:40.798842 (+ 2076us) server_negotiation.cc:407] Connection header received
0812 02:34:40.800147 (+ 1305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:34:40.800152 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:34:40.800222 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:34:40.800475 (+ 253us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:34:40.802141 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:40.803047 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:40.805090 (+ 2043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:40.805490 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:40.805610 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:40.809302 (+ 3692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:34:40.809307 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:34:40.809311 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:34:40.809348 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:34:40.811546 (+ 2198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:34:40.812419 (+ 873us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:34:40.812423 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:34:40.812425 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:34:40.812536 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:34:40.813014 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:34:40.813019 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:34:40.813023 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:34:40.813266 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:34:40.813360 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0812 02:34:40.815093 (+ 1733us) server_negotiation.cc:299] Negotiation successful
0812 02:34:40.815258 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":225,"thread_start_us":110,"threads_started":1}
I20260812 02:34:40.816334 28855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:40.796951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:35127 (local address 127.14.103.1:37489)
0812 02:34:40.798309 (+ 1358us) negotiation.cc:107] Waiting for socket to connect
0812 02:34:40.798380 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0812 02:34:40.799697 (+ 1317us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:34:40.800708 (+ 1011us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:34:40.800720 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:34:40.801131 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:34:40.801918 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:34:40.801942 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:34:40.803234 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:34:40.803238 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:34:40.804863 (+ 1625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:34:40.804883 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:34:40.805850 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:34:40.805857 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:34:40.806111 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:40.806999 (+ 888us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:34:40.807025 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:34:40.808990 (+ 1965us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:34:40.811744 (+ 2754us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:34:40.811753 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:34:40.811777 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:34:40.812254 (+ 477us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:34:40.812654 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:34:40.812659 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:34:40.812663 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:34:40.812895 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:34:40.813431 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:34:40.813439 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:34:40.813807 (+ 368us) client_negotiation.cc:769] Sending connection context
0812 02:34:40.814049 (+ 242us) client_negotiation.cc:240] Negotiation successful
0812 02:34:40.814458 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1091,"thread_start_us":161,"threads_started":1}
I20260812 02:34:40.818275 28852 heartbeater.cc:347] Connected to a master server at 127.14.103.62:35127
I20260812 02:34:40.818617 28852 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.103.2@KRBTEST.COM" created.
I20260812 02:34:40.819350 28852 heartbeater.cc:511] Master 127.14.103.62:35127 requested a full tablet report, sending...
I20260812 02:34:40.822808 28608 ts_manager.cc:194] Registered new tserver with Master: 3a6a0ec8e201413f8e57ba328d3e2f73 (127.14.103.1:42415)
I20260812 02:34:40.824661 28608 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.103.1@KRBTEST.COM'} at 127.14.103.1:37489
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.103.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.2.keytab.
Entry for principal kudu/127.14.103.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.2.keytab.
I20260812 02:34:40.843523 14748 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbep4M4/build/debug/bin/kudu
/tmp/dist-test-taskbep4M4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.103.20:59163/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.103.2:0
--local_ip_for_outbound_sockets=127.14.103.2
--webserver_interface=127.14.103.2
--webserver_port=0
--tserver_master_addrs=127.14.103.62:35127
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.2.keytab
--principal=kudu/127.14.103.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.103.20:45441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 02:34:40 dist-test-slave-2w3w krb5kdc[28062](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786502080, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260812 02:34:41.024964 28859 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:34:41.025415 28859 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:34:41.025578 28859 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:34:41.032406 28859 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:34:41.032517 28859 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:34:41.032610 28859 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.103.2
I20260812 02:34:41.041358 28859 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.103.20:45441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.14.103.20:59163/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.2.keytab
--principal=kudu/127.14.103.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.103.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.103.2
--webserver_port=0
--tserver_master_addrs=127.14.103.62:35127
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28859
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.103.2
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:34:41.043429 28859 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:34:41.045154 28859 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:34:41.063860 28866 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:34:41.064388 28869 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:34:41.064927 28865 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:34:41.066246 28859 server_base.cc:1034] running on GCE node
I20260812 02:34:41.066923 28859 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:34:41.067889 28859 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:34:41.081691 28859 hybrid_clock.cc:648] HybridClock initialized: now 1786502081081616 us; error 87 us; skew 500 ppm
Aug 12 02:34:41 dist-test-slave-2w3w krb5kdc[28062](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502081, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:34:41.086822 28859 init.cc:377] Logged in from keytab as kudu/127.14.103.2@KRBTEST.COM (short username kudu)
I20260812 02:34:41.088475 28859 webserver.cc:460] Webserver started at http://127.14.103.2:42591/ using document root <none> and password file <none>
I20260812 02:34:41.089529 28859 fs_manager.cc:359] Metadata directory not provided
I20260812 02:34:41.089660 28859 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:34:41.089933 28859 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 12 02:34:41 dist-test-slave-2w3w krb5kdc[28062](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502081, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.2@KRBTEST.COM for HTTP/127.14.103.20@KRBTEST.COM
I20260812 02:34:41.116496 28859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/minicluster-data/ts-1/data/instance:
uuid: "d22200f1ed91476e93061af36850e2bc"
format_stamp: "Formatted at 2026-08-12 02:34:41 on dist-test-slave-2w3w"
server_key: "0dea5b11d5e25feae49bc56bb93fc233"
server_key_iv: "11472b0f5ccf5b87fe989f049405deb3"
server_key_version: "kuduclusterkey@0"
I20260812 02:34:41.117429 28859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/minicluster-data/ts-1/wal/instance:
uuid: "d22200f1ed91476e93061af36850e2bc"
format_stamp: "Formatted at 2026-08-12 02:34:41 on dist-test-slave-2w3w"
server_key: "0dea5b11d5e25feae49bc56bb93fc233"
server_key_iv: "11472b0f5ccf5b87fe989f049405deb3"
server_key_version: "kuduclusterkey@0"
I20260812 02:34:41.130089 28859 fs_manager.cc:691] Time spent creating directory manager: real 0.012s user 0.002s sys 0.005s
I20260812 02:34:41.155778 28877 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:34:41.160991 28859 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.000s sys 0.004s
I20260812 02:34:41.161203 28859 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/minicluster-data/ts-1/wal
uuid: "d22200f1ed91476e93061af36850e2bc"
format_stamp: "Formatted at 2026-08-12 02:34:41 on dist-test-slave-2w3w"
server_key: "0dea5b11d5e25feae49bc56bb93fc233"
server_key_iv: "11472b0f5ccf5b87fe989f049405deb3"
server_key_version: "kuduclusterkey@0"
I20260812 02:34:41.161391 28859 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:34:41.208011 28859 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:34:41.212246 28859 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:34:41.212553 28859 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:34:41.213425 28859 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:34:41.214602 28859 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:34:41.214720 28859 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:34:41.214849 28859 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:34:41.214917 28859 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:34:41.227159 28859 rpc_server.cc:307] RPC server started. Bound to: 127.14.103.2:35243
I20260812 02:34:41.227576 29005 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.103.2:35243 every 8 connection(s)
I20260812 02:34:41.228596 28859 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/minicluster-data/ts-1/data/info.pb
I20260812 02:34:41.230484 14748 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbep4M4/build/debug/bin/kudu as pid 28859
I20260812 02:34:41.230619 14748 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/minicluster-data/ts-1/wal/instance
I20260812 02:34:41.230981 14748 external_mini_cluster.cc:1428] Setting key 27c0713bffc875c0ceb1ef419315e819
Aug 12 02:34:41 dist-test-slave-2w3w krb5kdc[28062](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502081, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.2@KRBTEST.COM for kudu/127.14.103.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.14.103.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.103.3@KRBTEST.COM" created.
I20260812 02:34:41.250615 28854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:41.231211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.2:56947 (local address 127.14.103.62:35127)
0812 02:34:41.231369 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0812 02:34:41.231373 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:34:41.234832 (+ 3459us) server_negotiation.cc:407] Connection header received
0812 02:34:41.236356 (+ 1524us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:34:41.236360 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:34:41.236425 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:34:41.236484 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:34:41.238384 (+ 1900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:41.239258 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:41.241071 (+ 1813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:41.241422 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:41.241515 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:41.245587 (+ 4072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:34:41.245592 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:34:41.245595 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:34:41.245630 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:34:41.247650 (+ 2020us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:34:41.248505 (+ 855us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:34:41.248509 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:34:41.248511 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:34:41.248617 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:34:41.249113 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:34:41.249117 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:34:41.249118 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:34:41.249376 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:34:41.249456 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0812 02:34:41.250244 (+ 788us) server_negotiation.cc:299] Negotiation successful
0812 02:34:41.250409 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260812 02:34:41.252357 29010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:41.232312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:35127 (local address 127.14.103.2:56947)
0812 02:34:41.234608 (+ 2296us) negotiation.cc:107] Waiting for socket to connect
0812 02:34:41.234685 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0812 02:34:41.236040 (+ 1355us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:34:41.237021 (+ 981us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:34:41.237032 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:34:41.237446 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:34:41.238186 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:34:41.238208 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:34:41.239442 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:34:41.239446 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:34:41.240900 (+ 1454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:34:41.240913 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:34:41.241756 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:34:41.241760 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:34:41.242000 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:41.243290 (+ 1290us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:34:41.243314 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:34:41.245285 (+ 1971us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:34:41.247842 (+ 2557us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:34:41.247852 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:34:41.247876 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:34:41.248349 (+ 473us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:34:41.248743 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:34:41.248746 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:34:41.248749 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:34:41.248979 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:34:41.249550 (+ 571us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:34:41.249557 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:34:41.249927 (+ 370us) client_negotiation.cc:769] Sending connection context
0812 02:34:41.250813 (+ 886us) client_negotiation.cc:240] Negotiation successful
0812 02:34:41.251248 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2030,"thread_start_us":190,"threads_started":1}
I20260812 02:34:41.254096 29008 heartbeater.cc:347] Connected to a master server at 127.14.103.62:35127
I20260812 02:34:41.254390 29008 heartbeater.cc:464] Registering TS with master...
I20260812 02:34:41.255213 29008 heartbeater.cc:511] Master 127.14.103.62:35127 requested a full tablet report, sending...
I20260812 02:34:41.256623 28608 ts_manager.cc:194] Registered new tserver with Master: d22200f1ed91476e93061af36850e2bc (127.14.103.2:35243)
I20260812 02:34:41.257449 28608 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.103.2@KRBTEST.COM'} at 127.14.103.2:56947
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.103.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.3.keytab.
Entry for principal kudu/127.14.103.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.3.keytab.
I20260812 02:34:41.276643 14748 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbep4M4/build/debug/bin/kudu
/tmp/dist-test-taskbep4M4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.103.20:59163/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.103.3:0
--local_ip_for_outbound_sockets=127.14.103.3
--webserver_interface=127.14.103.3
--webserver_port=0
--tserver_master_addrs=127.14.103.62:35127
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.3.keytab
--principal=kudu/127.14.103.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.103.20:45441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 02:34:41 dist-test-slave-2w3w krb5kdc[28062](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502080, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.62@KRBTEST.COM for HTTP/127.14.103.20@KRBTEST.COM
W20260812 02:34:41.456437 29015 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:34:41.456920 29015 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:34:41.457039 29015 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:34:41.463636 29015 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:34:41.463809 29015 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:34:41.463948 29015 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.103.3
I20260812 02:34:41.472536 29015 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.103.20:45441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.14.103.20:59163/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.3.keytab
--principal=kudu/127.14.103.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.103.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.103.3
--webserver_port=0
--tserver_master_addrs=127.14.103.62:35127
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29015
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.103.3
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:34:41.474264 29015 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:34:41.476166 29015 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:34:41.489545 29020 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:34:41.489560 29024 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:34:41.490859 29015 server_base.cc:1034] running on GCE node
W20260812 02:34:41.491104 29021 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:34:41.491683 29015 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:34:41.492574 29015 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:34:41.493741 29015 hybrid_clock.cc:648] HybridClock initialized: now 1786502081493711 us; error 68 us; skew 500 ppm
Aug 12 02:34:41 dist-test-slave-2w3w krb5kdc[28062](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502081, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:34:41.497982 29015 init.cc:377] Logged in from keytab as kudu/127.14.103.3@KRBTEST.COM (short username kudu)
I20260812 02:34:41.499336 29015 webserver.cc:460] Webserver started at http://127.14.103.3:46807/ using document root <none> and password file <none>
I20260812 02:34:41.500216 29015 fs_manager.cc:359] Metadata directory not provided
I20260812 02:34:41.500319 29015 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:34:41.500574 29015 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 12 02:34:41 dist-test-slave-2w3w krb5kdc[28062](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502081, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.3@KRBTEST.COM for HTTP/127.14.103.20@KRBTEST.COM
I20260812 02:34:41.519029 29015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/minicluster-data/ts-2/data/instance:
uuid: "c73ff27adfca4071a62812d783d9c872"
format_stamp: "Formatted at 2026-08-12 02:34:41 on dist-test-slave-2w3w"
server_key: "d91ced96da92d601a2e3828ed9945fb2"
server_key_iv: "c3f441406f4f181e38849abce7a9dd79"
server_key_version: "kuduclusterkey@0"
I20260812 02:34:41.520009 29015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/minicluster-data/ts-2/wal/instance:
uuid: "c73ff27adfca4071a62812d783d9c872"
format_stamp: "Formatted at 2026-08-12 02:34:41 on dist-test-slave-2w3w"
server_key: "d91ced96da92d601a2e3828ed9945fb2"
server_key_iv: "c3f441406f4f181e38849abce7a9dd79"
server_key_version: "kuduclusterkey@0"
I20260812 02:34:41.525967 29015 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260812 02:34:41.555675 29031 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:34:41.557919 29015 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.001s sys 0.000s
I20260812 02:34:41.558102 29015 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/minicluster-data/ts-2/wal
uuid: "c73ff27adfca4071a62812d783d9c872"
format_stamp: "Formatted at 2026-08-12 02:34:41 on dist-test-slave-2w3w"
server_key: "d91ced96da92d601a2e3828ed9945fb2"
server_key_iv: "c3f441406f4f181e38849abce7a9dd79"
server_key_version: "kuduclusterkey@0"
I20260812 02:34:41.558259 29015 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:34:41.574445 29015 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:34:41.578656 29015 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:34:41.578903 29015 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:34:41.579947 29015 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:34:41.581115 29015 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:34:41.581187 29015 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:34:41.581247 29015 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:34:41.581323 29015 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260812 02:34:41.593803 29015 rpc_server.cc:307] RPC server started. Bound to: 127.14.103.3:37645
I20260812 02:34:41.595264 29015 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/minicluster-data/ts-2/data/info.pb
I20260812 02:34:41.602861 14748 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbep4M4/build/debug/bin/kudu as pid 29015
I20260812 02:34:41.603051 14748 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/minicluster-data/ts-2/wal/instance
I20260812 02:34:41.603180 29157 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.103.3:37645 every 8 connection(s)
I20260812 02:34:41.603439 14748 external_mini_cluster.cc:1428] Setting key f336c7bcf0b8fc2b88c9a8a4f3be7598
Aug 12 02:34:41 dist-test-slave-2w3w krb5kdc[28062](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502081, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.3@KRBTEST.COM for kudu/127.14.103.62@KRBTEST.COM
I20260812 02:34:41.616317 28854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:41.599438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.3:35025 (local address 127.14.103.62:35127)
0812 02:34:41.599588 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0812 02:34:41.599593 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:34:41.599614 (+ 21us) server_negotiation.cc:407] Connection header received
0812 02:34:41.600357 (+ 743us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:34:41.600361 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:34:41.600433 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:34:41.600538 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:34:41.602627 (+ 2089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:41.603989 (+ 1362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:41.606216 (+ 2227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:41.606590 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:41.606674 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:41.611454 (+ 4780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:34:41.611459 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:34:41.611463 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:34:41.611500 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:34:41.613486 (+ 1986us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:34:41.614275 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:34:41.614280 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:34:41.614282 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:34:41.614386 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:34:41.614840 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:34:41.614844 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:34:41.614846 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:34:41.615149 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:34:41.615260 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0812 02:34:41.615974 (+ 714us) server_negotiation.cc:299] Negotiation successful
0812 02:34:41.616123 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260812 02:34:41.617817 29162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:41.598157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:35127 (local address 127.14.103.3:35025)
0812 02:34:41.598867 (+ 710us) negotiation.cc:107] Waiting for socket to connect
0812 02:34:41.598934 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0812 02:34:41.600073 (+ 1139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:34:41.600980 (+ 907us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:34:41.600994 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:34:41.601424 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:34:41.602240 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:34:41.602264 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:34:41.604229 (+ 1965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:34:41.604233 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:34:41.606017 (+ 1784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:34:41.606035 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:34:41.607190 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:34:41.607195 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:34:41.607443 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:41.608655 (+ 1212us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:34:41.608679 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:34:41.611132 (+ 2453us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:34:41.613674 (+ 2542us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:34:41.613683 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:34:41.613704 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:34:41.614134 (+ 430us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:34:41.614502 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:34:41.614505 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:34:41.614507 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:34:41.614712 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:34:41.615284 (+ 572us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:34:41.615292 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:34:41.615678 (+ 386us) client_negotiation.cc:769] Sending connection context
0812 02:34:41.615910 (+ 232us) client_negotiation.cc:240] Negotiation successful
0812 02:34:41.616287 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":218,"threads_started":1}
I20260812 02:34:41.619668 29158 heartbeater.cc:347] Connected to a master server at 127.14.103.62:35127
I20260812 02:34:41.619976 29158 heartbeater.cc:464] Registering TS with master...
I20260812 02:34:41.620772 29158 heartbeater.cc:511] Master 127.14.103.62:35127 requested a full tablet report, sending...
I20260812 02:34:41.622285 28607 ts_manager.cc:194] Registered new tserver with Master: c73ff27adfca4071a62812d783d9c872 (127.14.103.3:37645)
I20260812 02:34:41.623183 28607 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.103.3@KRBTEST.COM'} at 127.14.103.3:35025
I20260812 02:34:41.643640 14748 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 02:34:41.644456 14748 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbep4M4/build/debug/bin/kudu with pid 28701
I20260812 02:34:41.659125 14748 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbep4M4/build/debug/bin/kudu with pid 28859
I20260812 02:34:41.670451 14748 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbep4M4/build/debug/bin/kudu with pid 29015
I20260812 02:34:41.679704 14748 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbep4M4/build/debug/bin/kudu with pid 28571
I20260812 02:34:41.693471 14748 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbep4M4/build/debug/bin/kudu
/tmp/dist-test-taskbep4M4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.103.62:35127
--webserver_interface=127.14.103.62
--webserver_port=36527
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.62.keytab
--principal=kudu/127.14.103.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.103.20:45441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.103.62:35127
--ranger_config_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.103.20:59163/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.14.103.20:59163/kms with env {KRB5CCNAME=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:34:41.814924 29163 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:34:41.815268 29163 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:34:41.815313 29163 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:34:41.819576 29163 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:34:41.819669 29163 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:34:41.819689 29163 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:34:41.819706 29163 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:34:41.819722 29163 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 02:34:41.826050 29163 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.103.20:45441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.14.103.20:59163/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.14.103.62:35127
--ranger_config_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.62.keytab
--principal=kudu/127.14.103.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.103.62:35127
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.103.62
--webserver_port=36527
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29163
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:34:41.827322 29163 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:34:41.828378 29163 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:34:41.837641 29170 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:34:41.837646 29171 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:34:41.837687 29163 server_base.cc:1034] running on GCE node
W20260812 02:34:41.837646 29173 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:34:41.838557 29163 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:34:41.839785 29163 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:34:41.840967 29163 hybrid_clock.cc:648] HybridClock initialized: now 1786502081840946 us; error 37 us; skew 500 ppm
Aug 12 02:34:41 dist-test-slave-2w3w krb5kdc[28062](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502081, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:34:41.844193 29163 init.cc:377] Logged in from keytab as kudu/127.14.103.62@KRBTEST.COM (short username kudu)
I20260812 02:34:41.845255 29163 webserver.cc:460] Webserver started at http://127.14.103.62:36527/ using document root <none> and password file <none>
I20260812 02:34:41.845875 29163 fs_manager.cc:359] Metadata directory not provided
I20260812 02:34:41.845985 29163 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 12 02:34:41 dist-test-slave-2w3w krb5kdc[28062](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502081, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.62@KRBTEST.COM for HTTP/127.14.103.20@KRBTEST.COM
I20260812 02:34:41.926692 29163 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260812 02:34:41.929508 29182 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:34:41.931594 29163 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260812 02:34:41.931818 29163 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/minicluster-data/master-0/data,/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/minicluster-data/master-0/wal
uuid: "2b19b08f7ef849d698d640245912b1e4"
format_stamp: "Formatted at 2026-08-12 02:34:39 on dist-test-slave-2w3w"
server_key: "e4bc8128b5e5fc4ee11cae35cbd6a886"
server_key_iv: "0e98e13f00b9990507e8eca7de64f85a"
server_key_version: "kuduclusterkey@0"
I20260812 02:34:41.932485 29163 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:34:41.951740 29163 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:34:41.955513 29163 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:34:41.955864 29163 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:34:41.964663 29163 rpc_server.cc:307] RPC server started. Bound to: 127.14.103.62:35127
I20260812 02:34:41.964854 29239 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.103.62:35127 every 8 connection(s)
I20260812 02:34:41.966058 29163 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/minicluster-data/master-0/data/info.pb
I20260812 02:34:41.971172 29240 sys_catalog.cc:263] Verifying existing consensus state
I20260812 02:34:41.973050 29240 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b19b08f7ef849d698d640245912b1e4: Bootstrap starting.
I20260812 02:34:41.973852 14748 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbep4M4/build/debug/bin/kudu as pid 29163
I20260812 02:34:41.974485 14748 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbep4M4/build/debug/bin/kudu
/tmp/dist-test-taskbep4M4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.103.20:59163/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.103.1:42415
--local_ip_for_outbound_sockets=127.14.103.1
--tserver_master_addrs=127.14.103.62:35127
--webserver_port=33037
--webserver_interface=127.14.103.1
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.1.keytab
--principal=kudu/127.14.103.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.103.20:45441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.14.103.20:59163/kms with env {KRB5CCNAME=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 02:34:41.992733 29240 log.cc:824] T 00000000000000000000000000000000 P 2b19b08f7ef849d698d640245912b1e4: Log is configured to *not* fsync() on all Append() calls
I20260812 02:34:42.000777 29240 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b19b08f7ef849d698d640245912b1e4: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 02:34:42.001286 29240 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b19b08f7ef849d698d640245912b1e4: Bootstrap complete.
I20260812 02:34:42.006292 29240 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2b19b08f7ef849d698d640245912b1e4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b19b08f7ef849d698d640245912b1e4" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 35127 } }
I20260812 02:34:42.006878 29240 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2b19b08f7ef849d698d640245912b1e4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b19b08f7ef849d698d640245912b1e4, State: Initialized, Role: FOLLOWER
I20260812 02:34:42.007565 29240 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2b19b08f7ef849d698d640245912b1e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b19b08f7ef849d698d640245912b1e4" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 35127 } }
I20260812 02:34:42.007740 29240 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2b19b08f7ef849d698d640245912b1e4 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 02:34:42.007835 29240 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2b19b08f7ef849d698d640245912b1e4 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 02:34:42.007969 29240 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2b19b08f7ef849d698d640245912b1e4 [term 1 FOLLOWER]: Advancing to term 2
I20260812 02:34:42.010357 29240 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2b19b08f7ef849d698d640245912b1e4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b19b08f7ef849d698d640245912b1e4" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 35127 } }
I20260812 02:34:42.010751 29240 leader_election.cc:302] T 00000000000000000000000000000000 P 2b19b08f7ef849d698d640245912b1e4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2b19b08f7ef849d698d640245912b1e4; no voters:
I20260812 02:34:42.011503 29240 leader_election.cc:288] T 00000000000000000000000000000000 P 2b19b08f7ef849d698d640245912b1e4 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260812 02:34:42.011646 29246 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2b19b08f7ef849d698d640245912b1e4 [term 2 FOLLOWER]: Leader election won for term 2
I20260812 02:34:42.012168 29246 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2b19b08f7ef849d698d640245912b1e4 [term 2 LEADER]: Becoming Leader. State: Replica: 2b19b08f7ef849d698d640245912b1e4, State: Running, Role: LEADER
I20260812 02:34:42.012627 29246 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2b19b08f7ef849d698d640245912b1e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b19b08f7ef849d698d640245912b1e4" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 35127 } }
I20260812 02:34:42.013288 29240 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2b19b08f7ef849d698d640245912b1e4 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:34:42.014139 29247 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b19b08f7ef849d698d640245912b1e4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "2b19b08f7ef849d698d640245912b1e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b19b08f7ef849d698d640245912b1e4" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 35127 } } }
I20260812 02:34:42.014300 29247 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b19b08f7ef849d698d640245912b1e4 [sys.catalog]: This master's current role is: LEADER
I20260812 02:34:42.014617 29248 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b19b08f7ef849d698d640245912b1e4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2b19b08f7ef849d698d640245912b1e4. Latest consensus state: current_term: 2 leader_uuid: "2b19b08f7ef849d698d640245912b1e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b19b08f7ef849d698d640245912b1e4" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 35127 } } }
I20260812 02:34:42.014770 29248 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b19b08f7ef849d698d640245912b1e4 [sys.catalog]: This master's current role is: LEADER
I20260812 02:34:42.015336 29255 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 02:34:42.016141 29240 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260812 02:34:42.022696 29255 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 02:34:42.023960 29255 catalog_manager.cc:1259] Loaded cluster ID: a6819f31033d4d55a8c8065b815ee143
I20260812 02:34:42.024107 29255 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 02:34:42.027330 29255 catalog_manager.cc:1504] Loading token signing keys...
I20260812 02:34:42.028448 29255 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 2b19b08f7ef849d698d640245912b1e4: Loaded TSK: 0
I20260812 02:34:42.030117 29255 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260812 02:34:42.142499 29242 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:34:42.142912 29242 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:34:42.143006 29242 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:34:42.149778 29242 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:34:42.149964 29242 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:34:42.150086 29242 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.103.1
I20260812 02:34:42.159616 29242 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.103.20:45441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.14.103.20:59163/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.1.keytab
--principal=kudu/127.14.103.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.103.1:42415
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.103.1
--webserver_port=33037
--tserver_master_addrs=127.14.103.62:35127
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29242
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.103.1
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:34:42.162130 29242 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:34:42.164476 29242 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:34:42.180814 29289 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:34:42.180797 29285 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:34:42.181825 29284 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:34:42.182119 29242 server_base.cc:1034] running on GCE node
I20260812 02:34:42.182691 29242 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:34:42.183772 29242 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:34:42.187214 29242 hybrid_clock.cc:648] HybridClock initialized: now 1786502082187118 us; error 106 us; skew 500 ppm
Aug 12 02:34:42 dist-test-slave-2w3w krb5kdc[28062](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502082, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:34:42.192000 29242 init.cc:377] Logged in from keytab as kudu/127.14.103.1@KRBTEST.COM (short username kudu)
I20260812 02:34:42.193404 29242 webserver.cc:460] Webserver started at http://127.14.103.1:33037/ using document root <none> and password file <none>
I20260812 02:34:42.194223 29242 fs_manager.cc:359] Metadata directory not provided
I20260812 02:34:42.194373 29242 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 12 02:34:42 dist-test-slave-2w3w krb5kdc[28062](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502082, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.1@KRBTEST.COM for HTTP/127.14.103.20@KRBTEST.COM
I20260812 02:34:42.231410 29242 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260812 02:34:42.235306 29297 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:34:42.237149 29242 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260812 02:34:42.237309 29242 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/minicluster-data/ts-0/wal
uuid: "3a6a0ec8e201413f8e57ba328d3e2f73"
format_stamp: "Formatted at 2026-08-12 02:34:40 on dist-test-slave-2w3w"
server_key: "5c3d31e9c422523620f4edddfa66a22c"
server_key_iv: "c1720fc66ec2c24989828037627c3234"
server_key_version: "kuduclusterkey@0"
I20260812 02:34:42.238090 29242 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:34:42.263984 29242 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:34:42.267625 29242 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:34:42.267876 29242 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:34:42.268586 29242 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:34:42.269829 29242 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:34:42.269944 29242 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:34:42.270076 29242 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:34:42.270162 29242 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:34:42.283138 29242 rpc_server.cc:307] RPC server started. Bound to: 127.14.103.1:42415
I20260812 02:34:42.284809 29242 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/minicluster-data/ts-0/data/info.pb
I20260812 02:34:42.289700 29424 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.103.1:42415 every 8 connection(s)
I20260812 02:34:42.290174 14748 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbep4M4/build/debug/bin/kudu as pid 29242
I20260812 02:34:42.290745 14748 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbep4M4/build/debug/bin/kudu
/tmp/dist-test-taskbep4M4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.103.20:59163/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.103.2:35243
--local_ip_for_outbound_sockets=127.14.103.2
--tserver_master_addrs=127.14.103.62:35127
--webserver_port=42591
--webserver_interface=127.14.103.2
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.2.keytab
--principal=kudu/127.14.103.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.103.20:45441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.14.103.20:59163/kms with env {KRB5CCNAME=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 02:34:42 dist-test-slave-2w3w krb5kdc[28062](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502082, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.1@KRBTEST.COM for kudu/127.14.103.62@KRBTEST.COM
I20260812 02:34:42.315574 29427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:42.287656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:35127 (local address 127.14.103.1:48571)
0812 02:34:42.294503 (+ 6847us) negotiation.cc:107] Waiting for socket to connect
0812 02:34:42.294590 (+ 87us) client_negotiation.cc:174] Beginning negotiation
0812 02:34:42.295874 (+ 1284us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:34:42.298085 (+ 2211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:34:42.298096 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:34:42.298486 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:34:42.299254 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:34:42.299278 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:34:42.301606 (+ 2328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:34:42.301611 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:34:42.303382 (+ 1771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:34:42.303401 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:34:42.305004 (+ 1603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:34:42.305008 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:34:42.305232 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:42.306043 (+ 811us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:34:42.306066 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:34:42.307934 (+ 1868us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:34:42.311197 (+ 3263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:34:42.311205 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:34:42.311222 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:34:42.311621 (+ 399us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:34:42.312069 (+ 448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:34:42.312072 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:34:42.312074 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:34:42.312263 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:34:42.313258 (+ 995us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:34:42.313266 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:34:42.313588 (+ 322us) client_negotiation.cc:769] Sending connection context
0812 02:34:42.313783 (+ 195us) client_negotiation.cc:240] Negotiation successful
0812 02:34:42.314242 (+ 459us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6509,"thread_start_us":177,"threads_started":1}
I20260812 02:34:42.318349 29428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:42.289551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.1:48571 (local address 127.14.103.62:35127)
0812 02:34:42.296548 (+ 6997us) server_negotiation.cc:206] Beginning negotiation
0812 02:34:42.296574 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0812 02:34:42.296628 (+ 54us) server_negotiation.cc:407] Connection header received
0812 02:34:42.296993 (+ 365us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:34:42.297010 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:34:42.297506 (+ 496us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:34:42.299185 (+ 1679us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:34:42.299572 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:42.301371 (+ 1799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:42.303595 (+ 2224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:42.304534 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:42.304726 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:42.308250 (+ 3524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:34:42.308258 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:34:42.308277 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:34:42.308337 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:34:42.311005 (+ 2668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:34:42.311776 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:34:42.311785 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:34:42.311790 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:34:42.311955 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:34:42.312399 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:34:42.312405 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:34:42.312407 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:34:42.313021 (+ 614us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:34:42.313162 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0812 02:34:42.316928 (+ 3766us) server_negotiation.cc:299] Negotiation successful
0812 02:34:42.317285 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6769,"thread_start_us":180,"threads_started":1}
I20260812 02:34:42.321623 29425 heartbeater.cc:347] Connected to a master server at 127.14.103.62:35127
I20260812 02:34:42.321895 29425 heartbeater.cc:464] Registering TS with master...
I20260812 02:34:42.322518 29425 heartbeater.cc:511] Master 127.14.103.62:35127 requested a full tablet report, sending...
W20260812 02:34:42.325240 29425 heartbeater.cc:644] Failed to heartbeat to 127.14.103.62:35127 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260812 02:34:42.325419 29425 heartbeater.cc:464] Registering TS with master...
I20260812 02:34:42.325637 29425 heartbeater.cc:511] Master 127.14.103.62:35127 requested a full tablet report, sending...
I20260812 02:34:42.326934 29425 heartbeater.cc:464] Registering TS with master...
I20260812 02:34:42.327209 29425 heartbeater.cc:511] Master 127.14.103.62:35127 requested a full tablet report, sending...
W20260812 02:34:42.328177 29425 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260812 02:34:42.457015 29429 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:34:42.457383 29429 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:34:42.457475 29429 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:34:42.464725 29429 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:34:42.464847 29429 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:34:42.465075 29429 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.103.2
I20260812 02:34:42.474506 29429 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.103.20:45441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.14.103.20:59163/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.2.keytab
--principal=kudu/127.14.103.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.103.2:35243
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.103.2
--webserver_port=42591
--tserver_master_addrs=127.14.103.62:35127
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29429
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.103.2
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:34:42.476164 29429 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:34:42.477672 29429 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:34:42.494637 29435 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:34:42.495247 29437 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:34:42.497627 29434 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:34:42.498730 29429 server_base.cc:1034] running on GCE node
I20260812 02:34:42.499435 29429 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:34:42.500418 29429 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:34:42.503950 29429 hybrid_clock.cc:648] HybridClock initialized: now 1786502082503774 us; error 172 us; skew 500 ppm
Aug 12 02:34:42 dist-test-slave-2w3w krb5kdc[28062](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502082, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:34:42.508670 29429 init.cc:377] Logged in from keytab as kudu/127.14.103.2@KRBTEST.COM (short username kudu)
I20260812 02:34:42.512147 29429 webserver.cc:460] Webserver started at http://127.14.103.2:42591/ using document root <none> and password file <none>
I20260812 02:34:42.512861 29429 fs_manager.cc:359] Metadata directory not provided
I20260812 02:34:42.513000 29429 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 12 02:34:42 dist-test-slave-2w3w krb5kdc[28062](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502082, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.2@KRBTEST.COM for HTTP/127.14.103.20@KRBTEST.COM
I20260812 02:34:42.542276 29429 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.003s sys 0.000s
I20260812 02:34:42.547155 29447 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:34:42.548830 29429 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260812 02:34:42.549021 29429 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/minicluster-data/ts-1/wal
uuid: "d22200f1ed91476e93061af36850e2bc"
format_stamp: "Formatted at 2026-08-12 02:34:41 on dist-test-slave-2w3w"
server_key: "0dea5b11d5e25feae49bc56bb93fc233"
server_key_iv: "11472b0f5ccf5b87fe989f049405deb3"
server_key_version: "kuduclusterkey@0"
I20260812 02:34:42.549646 29429 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:34:42.564294 29429 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:34:42.569592 29429 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:34:42.569868 29429 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:34:42.570726 29429 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:34:42.572088 29429 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:34:42.572163 29429 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:34:42.572223 29429 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:34:42.572248 29429 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:34:42.584736 29429 rpc_server.cc:307] RPC server started. Bound to: 127.14.103.2:35243
I20260812 02:34:42.584795 29579 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.103.2:35243 every 8 connection(s)
I20260812 02:34:42.586225 29429 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/minicluster-data/ts-1/data/info.pb
I20260812 02:34:42.594519 14748 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbep4M4/build/debug/bin/kudu as pid 29429
I20260812 02:34:42.595109 14748 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbep4M4/build/debug/bin/kudu
/tmp/dist-test-taskbep4M4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.103.20:59163/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.103.3:37645
--local_ip_for_outbound_sockets=127.14.103.3
--tserver_master_addrs=127.14.103.62:35127
--webserver_port=46807
--webserver_interface=127.14.103.3
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.3.keytab
--principal=kudu/127.14.103.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.103.20:45441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.14.103.20:59163/kms with env {KRB5CCNAME=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 02:34:42 dist-test-slave-2w3w krb5kdc[28062](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502082, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.2@KRBTEST.COM for kudu/127.14.103.62@KRBTEST.COM
I20260812 02:34:42.605749 29582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:42.589008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:35127 (local address 127.14.103.2:45145)
0812 02:34:42.589524 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0812 02:34:42.589581 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0812 02:34:42.590565 (+ 984us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:34:42.591229 (+ 664us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:34:42.591238 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:34:42.591620 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:34:42.592196 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:34:42.592213 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:34:42.593099 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:34:42.593103 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:34:42.594429 (+ 1326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:34:42.594444 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:34:42.595516 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:34:42.595526 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:34:42.595799 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:42.596662 (+ 863us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:34:42.596691 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:34:42.598695 (+ 2004us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:34:42.601155 (+ 2460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:34:42.601164 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:34:42.601187 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:34:42.601809 (+ 622us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:34:42.602289 (+ 480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:34:42.602294 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:34:42.602296 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:34:42.602531 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:34:42.603177 (+ 646us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:34:42.603186 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:34:42.603635 (+ 449us) client_negotiation.cc:769] Sending connection context
0812 02:34:42.603927 (+ 292us) client_negotiation.cc:240] Negotiation successful
0812 02:34:42.604315 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":186,"threads_started":1}
I20260812 02:34:42.606828 29428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:42.588605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.2:45145 (local address 127.14.103.62:35127)
0812 02:34:42.588759 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0812 02:34:42.588764 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:34:42.589701 (+ 937us) server_negotiation.cc:407] Connection header received
0812 02:34:42.590808 (+ 1107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:34:42.590812 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:34:42.590877 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:34:42.591029 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:34:42.592378 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:42.592934 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:42.594618 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:42.595045 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:42.595195 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:42.599035 (+ 3840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:34:42.599040 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:34:42.599043 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:34:42.599079 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:34:42.600968 (+ 1889us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:34:42.602041 (+ 1073us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:34:42.602045 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:34:42.602047 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:34:42.602162 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:34:42.602666 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:34:42.602670 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:34:42.602672 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:34:42.602994 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:34:42.603098 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0812 02:34:42.606511 (+ 3413us) server_negotiation.cc:299] Negotiation successful
0812 02:34:42.606692 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260812 02:34:42.608916 29580 heartbeater.cc:347] Connected to a master server at 127.14.103.62:35127
I20260812 02:34:42.609265 29580 heartbeater.cc:464] Registering TS with master...
I20260812 02:34:42.610392 29580 heartbeater.cc:511] Master 127.14.103.62:35127 requested a full tablet report, sending...
W20260812 02:34:42.613579 29580 heartbeater.cc:644] Failed to heartbeat to 127.14.103.62:35127 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260812 02:34:42.613832 29580 heartbeater.cc:464] Registering TS with master...
I20260812 02:34:42.614215 29580 heartbeater.cc:511] Master 127.14.103.62:35127 requested a full tablet report, sending...
I20260812 02:34:42.615792 29580 heartbeater.cc:464] Registering TS with master...
I20260812 02:34:42.616092 29580 heartbeater.cc:511] Master 127.14.103.62:35127 requested a full tablet report, sending...
W20260812 02:34:42.617630 29580 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260812 02:34:42.756222 29583 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:34:42.756626 29583 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:34:42.756692 29583 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:34:42.762989 29583 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:34:42.763099 29583 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:34:42.763193 29583 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.103.3
I20260812 02:34:42.769609 29583 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.103.20:45441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.14.103.20:59163/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.3.keytab
--principal=kudu/127.14.103.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.103.3:37645
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.103.3
--webserver_port=46807
--tserver_master_addrs=127.14.103.62:35127
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29583
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.103.3
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:34:42.770834 29583 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:34:42.772217 29583 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:34:42.785498 29592 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:34:42.786432 29591 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:34:42.786787 29594 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:34:42.788357 29583 server_base.cc:1034] running on GCE node
I20260812 02:34:42.788981 29583 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:34:42.790647 29583 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:34:42.791947 29583 hybrid_clock.cc:648] HybridClock initialized: now 1786502082791888 us; error 73 us; skew 500 ppm
Aug 12 02:34:42 dist-test-slave-2w3w krb5kdc[28062](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502082, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:34:42.795624 29583 init.cc:377] Logged in from keytab as kudu/127.14.103.3@KRBTEST.COM (short username kudu)
I20260812 02:34:42.796904 29583 webserver.cc:460] Webserver started at http://127.14.103.3:46807/ using document root <none> and password file <none>
I20260812 02:34:42.797545 29583 fs_manager.cc:359] Metadata directory not provided
I20260812 02:34:42.797633 29583 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 12 02:34:42 dist-test-slave-2w3w krb5kdc[28062](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502082, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.3@KRBTEST.COM for HTTP/127.14.103.20@KRBTEST.COM
I20260812 02:34:42.824960 29583 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260812 02:34:42.829020 29602 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:34:42.831198 29583 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.005s sys 0.001s
I20260812 02:34:42.831431 29583 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/minicluster-data/ts-2/wal
uuid: "c73ff27adfca4071a62812d783d9c872"
format_stamp: "Formatted at 2026-08-12 02:34:41 on dist-test-slave-2w3w"
server_key: "d91ced96da92d601a2e3828ed9945fb2"
server_key_iv: "c3f441406f4f181e38849abce7a9dd79"
server_key_version: "kuduclusterkey@0"
I20260812 02:34:42.832140 29583 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:34:42.850054 29583 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:34:42.855401 29583 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:34:42.855674 29583 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:34:42.856534 29583 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:34:42.857764 29583 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:34:42.857848 29583 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:34:42.857910 29583 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:34:42.857947 29583 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:34:42.870663 29583 rpc_server.cc:307] RPC server started. Bound to: 127.14.103.3:37645
I20260812 02:34:42.870720 29730 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.103.3:37645 every 8 connection(s)
I20260812 02:34:42.872146 29583 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/minicluster-data/ts-2/data/info.pb
I20260812 02:34:42.876039 14748 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbep4M4/build/debug/bin/kudu as pid 29583
Aug 12 02:34:42 dist-test-slave-2w3w krb5kdc[28062](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502082, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.3@KRBTEST.COM for kudu/127.14.103.62@KRBTEST.COM
I20260812 02:34:42.888768 29737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:42.876829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57734 (local address 127.14.103.62:35127)
0812 02:34:42.877095 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0812 02:34:42.877099 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:34:42.877411 (+ 312us) server_negotiation.cc:407] Connection header received
0812 02:34:42.877651 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:34:42.877654 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:34:42.877703 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:34:42.877762 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:34:42.879000 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:42.879879 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:42.881146 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:42.881532 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:42.881666 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:42.883807 (+ 2141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:34:42.883813 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:34:42.883820 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:34:42.883849 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:34:42.886076 (+ 2227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:34:42.886707 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:34:42.886712 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:34:42.886714 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:34:42.886863 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:34:42.887274 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:34:42.887281 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:34:42.887283 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:34:42.887599 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:34:42.887842 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0812 02:34:42.888159 (+ 317us) server_negotiation.cc:299] Negotiation successful
0812 02:34:42.888588 (+ 429us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":194,"thread_start_us":95,"threads_started":1}
I20260812 02:34:42.891671 29428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:42.874519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.3:56697 (local address 127.14.103.62:35127)
0812 02:34:42.874671 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0812 02:34:42.874674 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 02:34:42.875999 (+ 1325us) server_negotiation.cc:407] Connection header received
0812 02:34:42.877406 (+ 1407us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:34:42.877412 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:34:42.877475 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:34:42.877577 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:34:42.879285 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:42.879819 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:42.882683 (+ 2864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:42.883062 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:42.883179 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:42.886552 (+ 3373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:34:42.886557 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:34:42.886560 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:34:42.886592 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:34:42.888505 (+ 1913us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:34:42.889243 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:34:42.889248 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:34:42.889253 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:34:42.889345 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:34:42.890027 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:34:42.890030 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:34:42.890032 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:34:42.890267 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:34:42.890366 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0812 02:34:42.891357 (+ 991us) server_negotiation.cc:299] Negotiation successful
0812 02:34:42.891499 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260812 02:34:42.894690 29736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:42.875116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:35127 (local address 127.14.103.3:56697)
0812 02:34:42.875752 (+ 636us) negotiation.cc:107] Waiting for socket to connect
0812 02:34:42.875824 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0812 02:34:42.877128 (+ 1304us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:34:42.877809 (+ 681us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:34:42.877819 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:34:42.878345 (+ 526us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:34:42.879130 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:34:42.879148 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:34:42.879967 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:34:42.879970 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:34:42.882515 (+ 2545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:34:42.882534 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:34:42.883407 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:34:42.883412 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:34:42.883661 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:42.884446 (+ 785us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:34:42.884470 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:34:42.886289 (+ 1819us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:34:42.888654 (+ 2365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:34:42.888663 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:34:42.888685 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:34:42.889121 (+ 436us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:34:42.889461 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:34:42.889465 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:34:42.889468 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:34:42.889888 (+ 420us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:34:42.890552 (+ 664us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:34:42.890560 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:34:42.891019 (+ 459us) client_negotiation.cc:769] Sending connection context
0812 02:34:42.891846 (+ 827us) client_negotiation.cc:240] Negotiation successful
0812 02:34:42.892407 (+ 561us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":185,"threads_started":1}
I20260812 02:34:42.896936 29731 heartbeater.cc:347] Connected to a master server at 127.14.103.62:35127
I20260812 02:34:42.897248 29731 heartbeater.cc:464] Registering TS with master...
I20260812 02:34:42.897979 29731 heartbeater.cc:511] Master 127.14.103.62:35127 requested a full tablet report, sending...
W20260812 02:34:42.900362 29731 heartbeater.cc:644] Failed to heartbeat to 127.14.103.62:35127 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260812 02:34:42.900532 29731 heartbeater.cc:464] Registering TS with master...
I20260812 02:34:42.900822 29731 heartbeater.cc:511] Master 127.14.103.62:35127 requested a full tablet report, sending...
I20260812 02:34:42.902109 29731 heartbeater.cc:464] Registering TS with master...
I20260812 02:34:42.903412 29731 heartbeater.cc:511] Master 127.14.103.62:35127 requested a full tablet report, sending...
I20260812 02:34:42.904542 29201 ts_manager.cc:194] Registered new tserver with Master: c73ff27adfca4071a62812d783d9c872 (127.14.103.3:37645)
I20260812 02:34:42.905757 29201 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.103.3@KRBTEST.COM'} at 127.14.103.3:56697
Aug 12 02:34:43 dist-test-slave-2w3w krb5kdc[28062](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786502083, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:34:43.328420 29425 heartbeater.cc:464] Registering TS with master...
I20260812 02:34:43.328764 29425 heartbeater.cc:511] Master 127.14.103.62:35127 requested a full tablet report, sending...
I20260812 02:34:43.329609 29201 ts_manager.cc:194] Registered new tserver with Master: 3a6a0ec8e201413f8e57ba328d3e2f73 (127.14.103.1:42415)
I20260812 02:34:43.330423 29201 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.103.1@KRBTEST.COM'} at 127.14.103.1:48571
I20260812 02:34:43.617930 29580 heartbeater.cc:464] Registering TS with master...
I20260812 02:34:43.618278 29580 heartbeater.cc:511] Master 127.14.103.62:35127 requested a full tablet report, sending...
I20260812 02:34:43.619279 29201 ts_manager.cc:194] Registered new tserver with Master: d22200f1ed91476e93061af36850e2bc (127.14.103.2:35243)
I20260812 02:34:43.620082 29201 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.103.2@KRBTEST.COM'} at 127.14.103.2:45145
I20260812 02:34:43.628769 14748 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 02:34:43.642094 29758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:43.631136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57750 (local address 127.14.103.62:35127)
0812 02:34:43.631452 (+ 316us) server_negotiation.cc:206] Beginning negotiation
0812 02:34:43.631457 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:34:43.631473 (+ 16us) server_negotiation.cc:407] Connection header received
0812 02:34:43.631520 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:34:43.631523 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:34:43.631573 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:34:43.631673 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:34:43.632782 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:43.633891 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:43.635270 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:43.635686 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:43.635813 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:43.637893 (+ 2080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:34:43.637897 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:34:43.637900 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:34:43.637938 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:34:43.639973 (+ 2035us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:34:43.640562 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:34:43.640566 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:34:43.640568 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:34:43.640670 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:34:43.641024 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:34:43.641028 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:34:43.641029 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:34:43.641289 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:34:43.641461 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0812 02:34:43.641746 (+ 285us) server_negotiation.cc:299] Negotiation successful
0812 02:34:43.641899 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":223,"thread_start_us":164,"threads_started":1}
I20260812 02:34:43.647986 29201 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57750:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 02:34:43.653999 29201 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 02:34:43.684068 29763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:43.671355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.2:35243 (local address 127.0.0.1:49610)
0812 02:34:43.675267 (+ 3912us) negotiation.cc:107] Waiting for socket to connect
0812 02:34:43.675339 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0812 02:34:43.675485 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:34:43.676315 (+ 830us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:34:43.676321 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:34:43.676344 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:34:43.676512 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:34:43.676521 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:34:43.679136 (+ 2615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:34:43.679144 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:34:43.680602 (+ 1458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:34:43.680616 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:34:43.683190 (+ 2574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:34:43.683194 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:34:43.683356 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:43.683414 (+ 58us) client_negotiation.cc:769] Sending connection context
0812 02:34:43.683517 (+ 103us) client_negotiation.cc:240] Negotiation successful
0812 02:34:43.683677 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3686,"mutex_wait_us":6,"thread_start_us":127,"threads_started":1}
I20260812 02:34:43.686636 29765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:43.674247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.3:37645 (local address 127.0.0.1:55658)
0812 02:34:43.676601 (+ 2354us) negotiation.cc:107] Waiting for socket to connect
0812 02:34:43.676617 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 02:34:43.676694 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:34:43.677566 (+ 872us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:34:43.677569 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:34:43.677587 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:34:43.677684 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:34:43.677690 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:34:43.679536 (+ 1846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:34:43.679540 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:34:43.681354 (+ 1814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:34:43.681366 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:34:43.685939 (+ 4573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:34:43.685944 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:34:43.686273 (+ 329us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:43.686303 (+ 30us) client_negotiation.cc:769] Sending connection context
0812 02:34:43.686372 (+ 69us) client_negotiation.cc:240] Negotiation successful
0812 02:34:43.686462 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2276,"thread_start_us":127,"threads_started":1}
I20260812 02:34:43.687544 29767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:43.674466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55658 (local address 127.14.103.3:37645)
0812 02:34:43.676959 (+ 2493us) server_negotiation.cc:206] Beginning negotiation
0812 02:34:43.676965 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 02:34:43.676989 (+ 24us) server_negotiation.cc:407] Connection header received
0812 02:34:43.677061 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:34:43.677066 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:34:43.677258 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:34:43.677405 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:34:43.677827 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:43.679115 (+ 1288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:43.683436 (+ 4321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:43.685779 (+ 2343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:43.686984 (+ 1205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:43.687187 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0812 02:34:43.687249 (+ 62us) server_negotiation.cc:299] Negotiation successful
0812 02:34:43.687372 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2377,"thread_start_us":82,"threads_started":1}
I20260812 02:34:43.689409 29764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:43.671951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49610 (local address 127.14.103.2:35243)
0812 02:34:43.675857 (+ 3906us) server_negotiation.cc:206] Beginning negotiation
0812 02:34:43.675866 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 02:34:43.675898 (+ 32us) server_negotiation.cc:407] Connection header received
0812 02:34:43.675964 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:34:43.675968 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:34:43.676145 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:34:43.676320 (+ 175us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:34:43.676659 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:43.678925 (+ 2266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:43.680832 (+ 1907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:43.683016 (+ 2184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:43.688862 (+ 5846us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:43.689059 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0812 02:34:43.689127 (+ 68us) server_negotiation.cc:299] Negotiation successful
0812 02:34:43.689246 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3768,"thread_start_us":146,"threads_started":1}
I20260812 02:34:43.691298 29657 tablet_service.cc:1467] Processing CreateTablet for tablet e2aab1a0a6074544bfe4161a1c2674c2 (DEFAULT_TABLE table=test-table [id=6f6195d30a70494da1708e5a8707fd93]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:34:43.692679 29657 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e2aab1a0a6074544bfe4161a1c2674c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:34:43.692704 29507 tablet_service.cc:1467] Processing CreateTablet for tablet e2aab1a0a6074544bfe4161a1c2674c2 (DEFAULT_TABLE table=test-table [id=6f6195d30a70494da1708e5a8707fd93]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:34:43.693956 29507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e2aab1a0a6074544bfe4161a1c2674c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:34:43.701092 29763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:43.675587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.1:42415 (local address 127.0.0.1:44596)
0812 02:34:43.684994 (+ 9407us) negotiation.cc:107] Waiting for socket to connect
0812 02:34:43.685013 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 02:34:43.685128 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:34:43.685990 (+ 862us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:34:43.685994 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:34:43.686015 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:34:43.686096 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:34:43.686102 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:34:43.689159 (+ 3057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:34:43.689162 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:34:43.690311 (+ 1149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:34:43.699028 (+ 8717us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:34:43.700592 (+ 1564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:34:43.700597 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:34:43.700715 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:43.700744 (+ 29us) client_negotiation.cc:769] Sending connection context
0812 02:34:43.700812 (+ 68us) client_negotiation.cc:240] Negotiation successful
0812 02:34:43.700917 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9323,"thread_start_us":106,"threads_started":1}
I20260812 02:34:43.703397 29768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:43.675274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44596 (local address 127.14.103.1:42415)
0812 02:34:43.679435 (+ 4161us) server_negotiation.cc:206] Beginning negotiation
0812 02:34:43.679442 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 02:34:43.685253 (+ 5811us) server_negotiation.cc:407] Connection header received
0812 02:34:43.685359 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:34:43.685366 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:34:43.685632 (+ 266us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:34:43.685804 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:34:43.687882 (+ 2078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:43.689011 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:43.699232 (+ 10221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:43.700213 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:43.700344 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:43.700504 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0812 02:34:43.703085 (+ 2581us) server_negotiation.cc:299] Negotiation successful
0812 02:34:43.703229 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4037,"thread_start_us":96,"threads_started":1}
I20260812 02:34:43.706876 29770 tablet_bootstrap.cc:492] T e2aab1a0a6074544bfe4161a1c2674c2 P c73ff27adfca4071a62812d783d9c872: Bootstrap starting.
I20260812 02:34:43.706991 29343 tablet_service.cc:1467] Processing CreateTablet for tablet e2aab1a0a6074544bfe4161a1c2674c2 (DEFAULT_TABLE table=test-table [id=6f6195d30a70494da1708e5a8707fd93]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:34:43.708317 29343 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e2aab1a0a6074544bfe4161a1c2674c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:34:43.709574 29770 tablet_bootstrap.cc:654] T e2aab1a0a6074544bfe4161a1c2674c2 P c73ff27adfca4071a62812d783d9c872: Neither blocks nor log segments found. Creating new log.
I20260812 02:34:43.710469 29770 log.cc:824] T e2aab1a0a6074544bfe4161a1c2674c2 P c73ff27adfca4071a62812d783d9c872: Log is configured to *not* fsync() on all Append() calls
I20260812 02:34:43.713004 29770 tablet_bootstrap.cc:492] T e2aab1a0a6074544bfe4161a1c2674c2 P c73ff27adfca4071a62812d783d9c872: No bootstrap required, opened a new log
I20260812 02:34:43.713267 29770 ts_tablet_manager.cc:1397] T e2aab1a0a6074544bfe4161a1c2674c2 P c73ff27adfca4071a62812d783d9c872: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260812 02:34:43.718092 29770 raft_consensus.cc:348] T e2aab1a0a6074544bfe4161a1c2674c2 P c73ff27adfca4071a62812d783d9c872 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d22200f1ed91476e93061af36850e2bc" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 35243 } } peers { permanent_uuid: "c73ff27adfca4071a62812d783d9c872" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 37645 } } peers { permanent_uuid: "3a6a0ec8e201413f8e57ba328d3e2f73" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 42415 } }
I20260812 02:34:43.718432 29770 raft_consensus.cc:374] T e2aab1a0a6074544bfe4161a1c2674c2 P c73ff27adfca4071a62812d783d9c872 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:34:43.718493 29770 raft_consensus.cc:729] T e2aab1a0a6074544bfe4161a1c2674c2 P c73ff27adfca4071a62812d783d9c872 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c73ff27adfca4071a62812d783d9c872, State: Initialized, Role: FOLLOWER
I20260812 02:34:43.719080 29770 consensus_queue.cc:260] T e2aab1a0a6074544bfe4161a1c2674c2 P c73ff27adfca4071a62812d783d9c872 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d22200f1ed91476e93061af36850e2bc" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 35243 } } peers { permanent_uuid: "c73ff27adfca4071a62812d783d9c872" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 37645 } } peers { permanent_uuid: "3a6a0ec8e201413f8e57ba328d3e2f73" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 42415 } }
I20260812 02:34:43.720292 29770 ts_tablet_manager.cc:1428] T e2aab1a0a6074544bfe4161a1c2674c2 P c73ff27adfca4071a62812d783d9c872: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260812 02:34:43.720726 29731 heartbeater.cc:503] Master 127.14.103.62:35127 was elected leader, sending a full tablet report...
I20260812 02:34:43.722505 29772 tablet_bootstrap.cc:492] T e2aab1a0a6074544bfe4161a1c2674c2 P d22200f1ed91476e93061af36850e2bc: Bootstrap starting.
I20260812 02:34:43.725306 29772 tablet_bootstrap.cc:654] T e2aab1a0a6074544bfe4161a1c2674c2 P d22200f1ed91476e93061af36850e2bc: Neither blocks nor log segments found. Creating new log.
I20260812 02:34:43.726367 29772 log.cc:824] T e2aab1a0a6074544bfe4161a1c2674c2 P d22200f1ed91476e93061af36850e2bc: Log is configured to *not* fsync() on all Append() calls
I20260812 02:34:43.729044 29772 tablet_bootstrap.cc:492] T e2aab1a0a6074544bfe4161a1c2674c2 P d22200f1ed91476e93061af36850e2bc: No bootstrap required, opened a new log
I20260812 02:34:43.729280 29772 ts_tablet_manager.cc:1397] T e2aab1a0a6074544bfe4161a1c2674c2 P d22200f1ed91476e93061af36850e2bc: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260812 02:34:43.729319 29775 tablet_bootstrap.cc:492] T e2aab1a0a6074544bfe4161a1c2674c2 P 3a6a0ec8e201413f8e57ba328d3e2f73: Bootstrap starting.
I20260812 02:34:43.734333 29775 tablet_bootstrap.cc:654] T e2aab1a0a6074544bfe4161a1c2674c2 P 3a6a0ec8e201413f8e57ba328d3e2f73: Neither blocks nor log segments found. Creating new log.
I20260812 02:34:43.735455 29775 log.cc:824] T e2aab1a0a6074544bfe4161a1c2674c2 P 3a6a0ec8e201413f8e57ba328d3e2f73: Log is configured to *not* fsync() on all Append() calls
I20260812 02:34:43.735517 29772 raft_consensus.cc:348] T e2aab1a0a6074544bfe4161a1c2674c2 P d22200f1ed91476e93061af36850e2bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d22200f1ed91476e93061af36850e2bc" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 35243 } } peers { permanent_uuid: "c73ff27adfca4071a62812d783d9c872" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 37645 } } peers { permanent_uuid: "3a6a0ec8e201413f8e57ba328d3e2f73" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 42415 } }
I20260812 02:34:43.735937 29772 raft_consensus.cc:374] T e2aab1a0a6074544bfe4161a1c2674c2 P d22200f1ed91476e93061af36850e2bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:34:43.736047 29772 raft_consensus.cc:729] T e2aab1a0a6074544bfe4161a1c2674c2 P d22200f1ed91476e93061af36850e2bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d22200f1ed91476e93061af36850e2bc, State: Initialized, Role: FOLLOWER
I20260812 02:34:43.736663 29772 consensus_queue.cc:260] T e2aab1a0a6074544bfe4161a1c2674c2 P d22200f1ed91476e93061af36850e2bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d22200f1ed91476e93061af36850e2bc" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 35243 } } peers { permanent_uuid: "c73ff27adfca4071a62812d783d9c872" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 37645 } } peers { permanent_uuid: "3a6a0ec8e201413f8e57ba328d3e2f73" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 42415 } }
I20260812 02:34:43.737663 29580 heartbeater.cc:503] Master 127.14.103.62:35127 was elected leader, sending a full tablet report...
I20260812 02:34:43.738065 29772 ts_tablet_manager.cc:1428] T e2aab1a0a6074544bfe4161a1c2674c2 P d22200f1ed91476e93061af36850e2bc: Time spent starting tablet: real 0.009s user 0.007s sys 0.000s
I20260812 02:34:43.741886 29775 tablet_bootstrap.cc:492] T e2aab1a0a6074544bfe4161a1c2674c2 P 3a6a0ec8e201413f8e57ba328d3e2f73: No bootstrap required, opened a new log
I20260812 02:34:43.742180 29775 ts_tablet_manager.cc:1397] T e2aab1a0a6074544bfe4161a1c2674c2 P 3a6a0ec8e201413f8e57ba328d3e2f73: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.000s
I20260812 02:34:43.746932 29775 raft_consensus.cc:348] T e2aab1a0a6074544bfe4161a1c2674c2 P 3a6a0ec8e201413f8e57ba328d3e2f73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d22200f1ed91476e93061af36850e2bc" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 35243 } } peers { permanent_uuid: "c73ff27adfca4071a62812d783d9c872" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 37645 } } peers { permanent_uuid: "3a6a0ec8e201413f8e57ba328d3e2f73" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 42415 } }
I20260812 02:34:43.747335 29775 raft_consensus.cc:374] T e2aab1a0a6074544bfe4161a1c2674c2 P 3a6a0ec8e201413f8e57ba328d3e2f73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:34:43.747447 29775 raft_consensus.cc:729] T e2aab1a0a6074544bfe4161a1c2674c2 P 3a6a0ec8e201413f8e57ba328d3e2f73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a6a0ec8e201413f8e57ba328d3e2f73, State: Initialized, Role: FOLLOWER
I20260812 02:34:43.748025 29775 consensus_queue.cc:260] T e2aab1a0a6074544bfe4161a1c2674c2 P 3a6a0ec8e201413f8e57ba328d3e2f73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d22200f1ed91476e93061af36850e2bc" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 35243 } } peers { permanent_uuid: "c73ff27adfca4071a62812d783d9c872" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 37645 } } peers { permanent_uuid: "3a6a0ec8e201413f8e57ba328d3e2f73" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 42415 } }
I20260812 02:34:43.748894 29425 heartbeater.cc:503] Master 127.14.103.62:35127 was elected leader, sending a full tablet report...
I20260812 02:34:43.749344 29775 ts_tablet_manager.cc:1428] T e2aab1a0a6074544bfe4161a1c2674c2 P 3a6a0ec8e201413f8e57ba328d3e2f73: Time spent starting tablet: real 0.007s user 0.002s sys 0.004s
I20260812 02:34:43.754390 29773 raft_consensus.cc:482] T e2aab1a0a6074544bfe4161a1c2674c2 P c73ff27adfca4071a62812d783d9c872 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 02:34:43.754611 29773 raft_consensus.cc:504] T e2aab1a0a6074544bfe4161a1c2674c2 P c73ff27adfca4071a62812d783d9c872 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d22200f1ed91476e93061af36850e2bc" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 35243 } } peers { permanent_uuid: "c73ff27adfca4071a62812d783d9c872" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 37645 } } peers { permanent_uuid: "3a6a0ec8e201413f8e57ba328d3e2f73" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 42415 } }
I20260812 02:34:43.756016 29773 leader_election.cc:288] T e2aab1a0a6074544bfe4161a1c2674c2 P c73ff27adfca4071a62812d783d9c872 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d22200f1ed91476e93061af36850e2bc (127.14.103.2:35243), 3a6a0ec8e201413f8e57ba328d3e2f73 (127.14.103.1:42415)
Aug 12 02:34:43 dist-test-slave-2w3w krb5kdc[28062](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502083, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.62@KRBTEST.COM for HTTP/127.14.103.20@KRBTEST.COM
I20260812 02:34:43.761670 29764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:43.757313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.3:39167 (local address 127.14.103.2:35243)
0812 02:34:43.757432 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0812 02:34:43.757436 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:34:43.757451 (+ 15us) server_negotiation.cc:407] Connection header received
0812 02:34:43.757505 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:34:43.757507 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:34:43.757548 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:34:43.757618 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:34:43.757964 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:43.758565 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:43.760201 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:43.760906 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:43.761023 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:43.761080 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0812 02:34:43.761441 (+ 361us) server_negotiation.cc:299] Negotiation successful
0812 02:34:43.761530 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260812 02:34:43.762472 29779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:43.756236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.2:35243 (local address 127.14.103.3:39167)
0812 02:34:43.756570 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0812 02:34:43.756588 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 02:34:43.756700 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:34:43.757634 (+ 934us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:34:43.757637 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:34:43.757655 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:34:43.757821 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:34:43.757828 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:34:43.758728 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:34:43.758731 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:34:43.760041 (+ 1310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:34:43.760052 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:34:43.761182 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:34:43.761185 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:34:43.761291 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:43.761315 (+ 24us) client_negotiation.cc:769] Sending connection context
0812 02:34:43.762047 (+ 732us) client_negotiation.cc:240] Negotiation successful
0812 02:34:43.762288 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":145,"threads_started":1}
I20260812 02:34:43.763509 29527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2aab1a0a6074544bfe4161a1c2674c2" candidate_uuid: "c73ff27adfca4071a62812d783d9c872" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d22200f1ed91476e93061af36850e2bc" is_pre_election: true
I20260812 02:34:43.763897 29527 raft_consensus.cc:2454] T e2aab1a0a6074544bfe4161a1c2674c2 P d22200f1ed91476e93061af36850e2bc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c73ff27adfca4071a62812d783d9c872 in term 0.
I20260812 02:34:43.764509 29604 leader_election.cc:302] T e2aab1a0a6074544bfe4161a1c2674c2 P c73ff27adfca4071a62812d783d9c872 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c73ff27adfca4071a62812d783d9c872, d22200f1ed91476e93061af36850e2bc; no voters:
I20260812 02:34:43.764860 29773 raft_consensus.cc:2781] T e2aab1a0a6074544bfe4161a1c2674c2 P c73ff27adfca4071a62812d783d9c872 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 02:34:43.764945 29773 raft_consensus.cc:482] T e2aab1a0a6074544bfe4161a1c2674c2 P c73ff27adfca4071a62812d783d9c872 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 02:34:43.764989 29773 raft_consensus.cc:3037] T e2aab1a0a6074544bfe4161a1c2674c2 P c73ff27adfca4071a62812d783d9c872 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:34:43.766505 29773 raft_consensus.cc:504] T e2aab1a0a6074544bfe4161a1c2674c2 P c73ff27adfca4071a62812d783d9c872 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d22200f1ed91476e93061af36850e2bc" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 35243 } } peers { permanent_uuid: "c73ff27adfca4071a62812d783d9c872" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 37645 } } peers { permanent_uuid: "3a6a0ec8e201413f8e57ba328d3e2f73" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 42415 } }
I20260812 02:34:43.766944 29773 leader_election.cc:288] T e2aab1a0a6074544bfe4161a1c2674c2 P c73ff27adfca4071a62812d783d9c872 [CANDIDATE]: Term 1 election: Requested vote from peers d22200f1ed91476e93061af36850e2bc (127.14.103.2:35243), 3a6a0ec8e201413f8e57ba328d3e2f73 (127.14.103.1:42415)
I20260812 02:34:43.767848 29527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2aab1a0a6074544bfe4161a1c2674c2" candidate_uuid: "c73ff27adfca4071a62812d783d9c872" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d22200f1ed91476e93061af36850e2bc"
I20260812 02:34:43.768014 29527 raft_consensus.cc:3037] T e2aab1a0a6074544bfe4161a1c2674c2 P d22200f1ed91476e93061af36850e2bc [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:34:43.769559 29527 raft_consensus.cc:2454] T e2aab1a0a6074544bfe4161a1c2674c2 P d22200f1ed91476e93061af36850e2bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c73ff27adfca4071a62812d783d9c872 in term 1.
I20260812 02:34:43.770057 29604 leader_election.cc:302] T e2aab1a0a6074544bfe4161a1c2674c2 P c73ff27adfca4071a62812d783d9c872 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c73ff27adfca4071a62812d783d9c872, d22200f1ed91476e93061af36850e2bc; no voters:
I20260812 02:34:43.770340 29773 raft_consensus.cc:2781] T e2aab1a0a6074544bfe4161a1c2674c2 P c73ff27adfca4071a62812d783d9c872 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:34:43.770568 29773 raft_consensus.cc:686] T e2aab1a0a6074544bfe4161a1c2674c2 P c73ff27adfca4071a62812d783d9c872 [term 1 LEADER]: Becoming Leader. State: Replica: c73ff27adfca4071a62812d783d9c872, State: Running, Role: LEADER
I20260812 02:34:43.770991 29773 consensus_queue.cc:237] T e2aab1a0a6074544bfe4161a1c2674c2 P c73ff27adfca4071a62812d783d9c872 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d22200f1ed91476e93061af36850e2bc" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 35243 } } peers { permanent_uuid: "c73ff27adfca4071a62812d783d9c872" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 37645 } } peers { permanent_uuid: "3a6a0ec8e201413f8e57ba328d3e2f73" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 42415 } }
I20260812 02:34:43.771782 29768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:43.757174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.3:50763 (local address 127.14.103.1:42415)
0812 02:34:43.757295 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0812 02:34:43.757300 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:34:43.757314 (+ 14us) server_negotiation.cc:407] Connection header received
0812 02:34:43.757371 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:34:43.757375 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:34:43.757422 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:34:43.757498 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:34:43.761202 (+ 3704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:43.761925 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:43.770145 (+ 8220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:43.770810 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:43.770921 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:43.771006 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0812 02:34:43.771526 (+ 520us) server_negotiation.cc:299] Negotiation successful
0812 02:34:43.771621 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260812 02:34:43.772295 29780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:43.756690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.1:42415 (local address 127.14.103.3:50763)
0812 02:34:43.757005 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0812 02:34:43.757023 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 02:34:43.757119 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:34:43.759061 (+ 1942us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:34:43.759064 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:34:43.759086 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:34:43.761038 (+ 1952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:34:43.761047 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:34:43.762074 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:34:43.762076 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:34:43.769960 (+ 7884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:34:43.769977 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:34:43.771200 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:34:43.771212 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:34:43.771334 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:43.771363 (+ 29us) client_negotiation.cc:769] Sending connection context
0812 02:34:43.772050 (+ 687us) client_negotiation.cc:240] Negotiation successful
0812 02:34:43.772147 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":111,"threads_started":1}
I20260812 02:34:43.773254 29357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2aab1a0a6074544bfe4161a1c2674c2" candidate_uuid: "c73ff27adfca4071a62812d783d9c872" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a6a0ec8e201413f8e57ba328d3e2f73" is_pre_election: true
I20260812 02:34:43.773561 29357 raft_consensus.cc:2454] T e2aab1a0a6074544bfe4161a1c2674c2 P 3a6a0ec8e201413f8e57ba328d3e2f73 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c73ff27adfca4071a62812d783d9c872 in term 0.
I20260812 02:34:43.773904 29375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2aab1a0a6074544bfe4161a1c2674c2" candidate_uuid: "c73ff27adfca4071a62812d783d9c872" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a6a0ec8e201413f8e57ba328d3e2f73"
I20260812 02:34:43.774019 29375 raft_consensus.cc:3037] T e2aab1a0a6074544bfe4161a1c2674c2 P 3a6a0ec8e201413f8e57ba328d3e2f73 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:34:43.775812 29201 catalog_manager.cc:5495] T e2aab1a0a6074544bfe4161a1c2674c2 P c73ff27adfca4071a62812d783d9c872 reported cstate change: term changed from 0 to 1, leader changed from <none> to c73ff27adfca4071a62812d783d9c872 (127.14.103.3). New cstate: current_term: 1 leader_uuid: "c73ff27adfca4071a62812d783d9c872" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d22200f1ed91476e93061af36850e2bc" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 35243 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c73ff27adfca4071a62812d783d9c872" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 37645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3a6a0ec8e201413f8e57ba328d3e2f73" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 42415 } health_report { overall_health: UNKNOWN } } }
I20260812 02:34:43.778404 29375 raft_consensus.cc:2454] T e2aab1a0a6074544bfe4161a1c2674c2 P 3a6a0ec8e201413f8e57ba328d3e2f73 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c73ff27adfca4071a62812d783d9c872 in term 1.
I20260812 02:34:43.790747 29767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:43.786581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55670 (local address 127.14.103.3:37645)
0812 02:34:43.786693 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0812 02:34:43.786698 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:34:43.786719 (+ 21us) server_negotiation.cc:407] Connection header received
0812 02:34:43.786795 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:34:43.786798 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:34:43.786870 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:34:43.787021 (+ 151us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 02:34:43.787296 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:43.787838 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:43.788835 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:43.789144 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:43.789256 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:43.789682 (+ 426us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 02:34:43.790268 (+ 586us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 02:34:43.790400 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0812 02:34:43.790502 (+ 102us) server_negotiation.cc:299] Negotiation successful
0812 02:34:43.790585 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
W20260812 02:34:43.796999 29426 tablet.cc:2369] T e2aab1a0a6074544bfe4161a1c2674c2 P 3a6a0ec8e201413f8e57ba328d3e2f73: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 02:34:43.800642 29375 raft_consensus.cc:1261] T e2aab1a0a6074544bfe4161a1c2674c2 P 3a6a0ec8e201413f8e57ba328d3e2f73 [term 1 FOLLOWER]: Refusing update from remote peer c73ff27adfca4071a62812d783d9c872: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:34:43.801757 29527 raft_consensus.cc:1261] T e2aab1a0a6074544bfe4161a1c2674c2 P d22200f1ed91476e93061af36850e2bc [term 1 FOLLOWER]: Refusing update from remote peer c73ff27adfca4071a62812d783d9c872: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:34:43.802906 29773 consensus_queue.cc:1035] T e2aab1a0a6074544bfe4161a1c2674c2 P c73ff27adfca4071a62812d783d9c872 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d22200f1ed91476e93061af36850e2bc" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 35243 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:34:43.803159 29781 consensus_queue.cc:1035] T e2aab1a0a6074544bfe4161a1c2674c2 P c73ff27adfca4071a62812d783d9c872 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a6a0ec8e201413f8e57ba328d3e2f73" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 42415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:34:43.837010 29785 mvcc.cc:205] Tried to move back new op lower bound from 7317512535230980096 to 7317512535131017216. Current Snapshot: MvccSnapshot[applied={T|T < 7317512535230980096}]
I20260812 02:34:43.843459 29201 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57750:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 02:34:43.843695 29201 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57750:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 02:34:43.845050 29786 mvcc.cc:205] Tried to move back new op lower bound from 7317512535230980096 to 7317512535131017216. Current Snapshot: MvccSnapshot[applied={T|T < 7317512535230980096}]
I20260812 02:34:43.847152 29201 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2b19b08f7ef849d698d640245912b1e4: Sending DeleteTablet for 3 replicas of tablet e2aab1a0a6074544bfe4161a1c2674c2
I20260812 02:34:43.848480 29507 tablet_service.cc:1514] Processing DeleteTablet for tablet e2aab1a0a6074544bfe4161a1c2674c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:34:43 UTC) from {username='kudu', principal='kudu/127.14.103.62@KRBTEST.COM'} at 127.0.0.1:49610
I20260812 02:34:43.849189 29657 tablet_service.cc:1514] Processing DeleteTablet for tablet e2aab1a0a6074544bfe4161a1c2674c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:34:43 UTC) from {username='kudu', principal='kudu/127.14.103.62@KRBTEST.COM'} at 127.0.0.1:55658
I20260812 02:34:43.849900 29343 tablet_service.cc:1514] Processing DeleteTablet for tablet e2aab1a0a6074544bfe4161a1c2674c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:34:43 UTC) from {username='kudu', principal='kudu/127.14.103.62@KRBTEST.COM'} at 127.0.0.1:44596
I20260812 02:34:43.851003 14748 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbep4M4/build/debug/bin/kudu with pid 29242
I20260812 02:34:43.862756 29795 tablet_replica.cc:330] T e2aab1a0a6074544bfe4161a1c2674c2 P c73ff27adfca4071a62812d783d9c872: stopping tablet replica
I20260812 02:34:43.869277 29795 raft_consensus.cc:2229] T e2aab1a0a6074544bfe4161a1c2674c2 P c73ff27adfca4071a62812d783d9c872 [term 1 LEADER]: Raft consensus shutting down.
I20260812 02:34:43.869750 29794 tablet_replica.cc:330] T e2aab1a0a6074544bfe4161a1c2674c2 P d22200f1ed91476e93061af36850e2bc: stopping tablet replica
I20260812 02:34:43.870180 29794 raft_consensus.cc:2229] T e2aab1a0a6074544bfe4161a1c2674c2 P d22200f1ed91476e93061af36850e2bc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 02:34:43.887393 29795 raft_consensus.cc:2258] T e2aab1a0a6074544bfe4161a1c2674c2 P c73ff27adfca4071a62812d783d9c872 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:34:43.887991 29794 raft_consensus.cc:2258] T e2aab1a0a6074544bfe4161a1c2674c2 P d22200f1ed91476e93061af36850e2bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:34:43.890048 29795 ts_tablet_manager.cc:1905] T e2aab1a0a6074544bfe4161a1c2674c2 P c73ff27adfca4071a62812d783d9c872: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:34:43.891888 29794 ts_tablet_manager.cc:1905] T e2aab1a0a6074544bfe4161a1c2674c2 P d22200f1ed91476e93061af36850e2bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:34:43.897019 29795 ts_tablet_manager.cc:1918] T e2aab1a0a6074544bfe4161a1c2674c2 P c73ff27adfca4071a62812d783d9c872: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 02:34:43.898488 29794 ts_tablet_manager.cc:1918] T e2aab1a0a6074544bfe4161a1c2674c2 P d22200f1ed91476e93061af36850e2bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
W20260812 02:34:43.899006 29184 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.103.1:42415 (error 108)
I20260812 02:34:43.900139 29794 log.cc:1196] T e2aab1a0a6074544bfe4161a1c2674c2 P d22200f1ed91476e93061af36850e2bc: Deleting WAL directory at /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/minicluster-data/ts-1/wal/wals/e2aab1a0a6074544bfe4161a1c2674c2
I20260812 02:34:43.900434 14748 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbep4M4/build/debug/bin/kudu with pid 29429
I20260812 02:34:43.905486 29795 log.cc:1196] T e2aab1a0a6074544bfe4161a1c2674c2 P c73ff27adfca4071a62812d783d9c872: Deleting WAL directory at /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786501898713699-14748-0/minicluster-data/ts-2/wal/wals/e2aab1a0a6074544bfe4161a1c2674c2
I20260812 02:34:43.911290 29795 ts_tablet_manager.cc:1939] T e2aab1a0a6074544bfe4161a1c2674c2 P c73ff27adfca4071a62812d783d9c872: Deleting consensus metadata
I20260812 02:34:43.913034 29186 catalog_manager.cc:4841] TS c73ff27adfca4071a62812d783d9c872 (127.14.103.3:37645): tablet e2aab1a0a6074544bfe4161a1c2674c2 (table test-table [id=6f6195d30a70494da1708e5a8707fd93]) successfully deleted
I20260812 02:34:43.914474 29763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:43.914117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.1:42415 (local address 127.0.0.1:44602)
0812 02:34:43.914245 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0812 02:34:43.914349 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.1:42415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260812 02:34:43.915055 29184 catalog_manager.cc:4584] TS 3a6a0ec8e201413f8e57ba328d3e2f73 (127.14.103.1:42415): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e2aab1a0a6074544bfe4161a1c2674c2: Network error: Client connection negotiation failed: client connection to 127.14.103.1:42415: connect: Connection refused (error 111)
W20260812 02:34:43.917568 29763 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:34:43.916968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.2:35243 (local address 127.0.0.1:49624)
0812 02:34:43.917139 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0812 02:34:43.917154 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 02:34:43.917243 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:34:43.917459 (+ 216us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.2:35243: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":99}
I20260812 02:34:43.918445 14748 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbep4M4/build/debug/bin/kudu with pid 29583
I20260812 02:34:43.928970 14748 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbep4M4/build/debug/bin/kudu with pid 29163
I20260812 02:34:43.942807 14748 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260812 02:34:44.289498 14748 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260812 02:34:44.289649 14748 mini_ranger.cc:66] Stopping Ranger...
I20260812 02:34:44.646375 14748 mini_ranger.cc:68] Stopped Ranger
2026-08-12 02:34:44.646 UTC [28098] LOG: received smart shutdown request
2026-08-12 02:34:44.654 UTC [28098] LOG: background worker "logical replication launcher" (PID 28108) exited with exit code 1
2026-08-12 02:34:44.654 UTC [28103] LOG: shutting down
2026-08-12 02:34:44.902 UTC [28098] LOG: database system is shut down
2026-08-12T02:34:44Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (58515 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786501898713699-14748-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 02:34:45 dist-test-slave-2w3w krb5kdc[29807](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 02:34:45 dist-test-slave-2w3w krb5kdc[29807](info): set up 2 sockets
Aug 12 02:34:45 dist-test-slave-2w3w krb5kdc[29807](info): commencing operation
krb5kdc: starting...
W20260812 02:34:46.201506 14748 mini_kdc.cc:121] Time spent starting KDC: real 1.132s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786501898713699-14748-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786501898713699-14748-0/krb5kdc/test-user.keytab.
Aug 12 02:34:46 dist-test-slave-2w3w krb5kdc[29807](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502086, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T02:34:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T02:34:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.103.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.103.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.103.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.62.keytab.
Entry for principal kudu/127.14.103.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.62.keytab.
I20260812 02:34:46.348078 14748 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbep4M4/build/debug/bin/kudu
/tmp/dist-test-taskbep4M4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786501898713699-14748-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786501898713699-14748-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786501898713699-14748-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786501898713699-14748-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.103.62:35903
--webserver_interface=127.14.103.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.62.keytab
--principal=kudu/127.14.103.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.103.20:33271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.103.62:35903,127.14.103.61:44111,127.14.103.60:44099
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786501898713699-14748-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786501898713699-14748-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786501898713699-14748-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:34:46.465514 29822 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:34:46.465876 29822 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:34:46.465967 29822 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:34:46.470126 29822 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:34:46.470243 29822 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:34:46.470286 29822 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:34:46.470342 29822 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:34:46.470383 29822 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 02:34:46.476075 29822 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.103.20:33271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786501898713699-14748-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786501898713699-14748-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.103.62:35903,127.14.103.61:44111,127.14.103.60:44099
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.62.keytab
--principal=kudu/127.14.103.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.103.62:35903
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786501898713699-14748-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.103.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29822
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786501898713699-14748-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:34:46.477248 29822 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:34:46.478381 29822 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:34:46.487210 29827 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:34:46.487210 29832 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:34:46.487210 29829 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:34:46.487514 29822 server_base.cc:1034] running on GCE node
I20260812 02:34:46.488317 29822 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:34:46.489609 29822 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:34:46.490818 29822 hybrid_clock.cc:648] HybridClock initialized: now 1786502086490793 us; error 41 us; skew 500 ppm
Aug 12 02:34:46 dist-test-slave-2w3w krb5kdc[29807](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502086, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:34:46.494182 29822 init.cc:377] Logged in from keytab as kudu/127.14.103.62@KRBTEST.COM (short username kudu)
I20260812 02:34:46.495325 29822 webserver.cc:460] Webserver started at http://127.14.103.62:43255/ using document root <none> and password file <none>
I20260812 02:34:46.495908 29822 fs_manager.cc:359] Metadata directory not provided
I20260812 02:34:46.496001 29822 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:34:46.496235 29822 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:34:46.498365 29822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786501898713699-14748-0/minicluster-data/master-0/data/instance:
uuid: "915d272f1ca84bbbb64f7a710f4482f2"
format_stamp: "Formatted at 2026-08-12 02:34:46 on dist-test-slave-2w3w"
server_key: "c454f6a23cd50b786d6337066e690fe1"
server_key_iv: "302ee9bcc7443f2bf5be16f97607cb01"
server_key_version: "encryptionkey@0"
I20260812 02:34:46.498991 29822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786501898713699-14748-0/minicluster-data/master-0/wal/instance:
uuid: "915d272f1ca84bbbb64f7a710f4482f2"
format_stamp: "Formatted at 2026-08-12 02:34:46 on dist-test-slave-2w3w"
server_key: "c454f6a23cd50b786d6337066e690fe1"
server_key_iv: "302ee9bcc7443f2bf5be16f97607cb01"
server_key_version: "encryptionkey@0"
I20260812 02:34:46.502949 29822 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260812 02:34:46.505741 29839 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:34:46.507174 29822 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 02:34:46.507329 29822 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786501898713699-14748-0/minicluster-data/master-0/data,/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786501898713699-14748-0/minicluster-data/master-0/wal
uuid: "915d272f1ca84bbbb64f7a710f4482f2"
format_stamp: "Formatted at 2026-08-12 02:34:46 on dist-test-slave-2w3w"
server_key: "c454f6a23cd50b786d6337066e690fe1"
server_key_iv: "302ee9bcc7443f2bf5be16f97607cb01"
server_key_version: "encryptionkey@0"
I20260812 02:34:46.507472 29822 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786501898713699-14748-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786501898713699-14748-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786501898713699-14748-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:34:46.527478 29822 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:34:46.531484 29822 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:34:46.531754 29822 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:34:46.540848 29822 rpc_server.cc:307] RPC server started. Bound to: 127.14.103.62:35903
I20260812 02:34:46.540874 29899 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.103.62:35903 every 8 connection(s)
I20260812 02:34:46.541951 29822 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786501898713699-14748-0/minicluster-data/master-0/data/info.pb
I20260812 02:34:46.544996 29900 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:34:46.546300 14748 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbep4M4/build/debug/bin/kudu as pid 29822
I20260812 02:34:46.546411 14748 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786501898713699-14748-0/minicluster-data/master-0/wal/instance
I20260812 02:34:46.546721 14748 external_mini_cluster.cc:1428] Setting key ee7edc8816ff215247491d2c444325cb
I20260812 02:34:46.549194 29900 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.103.62" port: 35903 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.14.103.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.103.61@KRBTEST.COM" created.
Aug 12 02:34:46 dist-test-slave-2w3w krb5kdc[29807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502086, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.62@KRBTEST.COM for kudu/127.14.103.62@KRBTEST.COM
I20260812 02:34:46.569236 29904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:46.552924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42154 (local address 127.14.103.62:35903)
0812 02:34:46.553321 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0812 02:34:46.553335 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0812 02:34:46.553391 (+ 56us) server_negotiation.cc:407] Connection header received
0812 02:34:46.553925 (+ 534us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:34:46.553935 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:34:46.554308 (+ 373us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:34:46.554559 (+ 251us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:34:46.555828 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:46.557431 (+ 1603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:46.558680 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:46.559479 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:46.559624 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:46.563082 (+ 3458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:34:46.563091 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:34:46.563105 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:34:46.563134 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:34:46.565681 (+ 2547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:34:46.566374 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:34:46.566380 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:34:46.566381 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:34:46.566506 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:34:46.566888 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:34:46.566890 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:34:46.566891 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:34:46.567388 (+ 497us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:34:46.567501 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0812 02:34:46.568024 (+ 523us) server_negotiation.cc:299] Negotiation successful
0812 02:34:46.568377 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":52,"server-negotiator.queue_time_us":245,"thread_start_us":107,"threads_started":1}
I20260812 02:34:46.569245 29903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:46.551957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:35903 (local address 127.0.0.1:42154)
0812 02:34:46.552502 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0812 02:34:46.552558 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0812 02:34:46.553572 (+ 1014us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:34:46.554593 (+ 1021us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:34:46.554598 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:34:46.555134 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:34:46.555643 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:34:46.555659 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:34:46.557598 (+ 1939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:34:46.557601 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:34:46.558503 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:34:46.558515 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:34:46.559660 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:34:46.559663 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:34:46.559895 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:46.560612 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:34:46.560630 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:34:46.562615 (+ 1985us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:34:46.565824 (+ 3209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:34:46.565839 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:34:46.565854 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:34:46.566228 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:34:46.566604 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:34:46.566610 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:34:46.566615 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:34:46.566788 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:34:46.567518 (+ 730us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:34:46.567528 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:34:46.567737 (+ 209us) client_negotiation.cc:769] Sending connection context
0812 02:34:46.567961 (+ 224us) client_negotiation.cc:240] Negotiation successful
0812 02:34:46.568489 (+ 528us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"mutex_wait_us":208,"thread_start_us":162,"threads_started":1}
I20260812 02:34:46.572163 29900 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.103.61" port: 44111 } has no permanent_uuid. Determining permanent_uuid...
I20260812 02:34:46.573125 29903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:46.572778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.61:44111 (local address 127.0.0.1:57332)
0812 02:34:46.572960 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0812 02:34:46.573044 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.61:44111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260812 02:34:46.573395 29843 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.103.61:44111: connect: Connection refused (error 111)
I20260812 02:34:46.574338 29903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:46.574102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.61:44111 (local address 127.0.0.1:57338)
0812 02:34:46.574222 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0812 02:34:46.574272 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.61:44111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260812 02:34:46.574523 29900 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.103.61:44111: Network error: Client connection negotiation failed: client connection to 127.14.103.61:44111: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.103.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.61.keytab.
Entry for principal kudu/127.14.103.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.61.keytab.
I20260812 02:34:46.582432 14748 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbep4M4/build/debug/bin/kudu
/tmp/dist-test-taskbep4M4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786501898713699-14748-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786501898713699-14748-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786501898713699-14748-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786501898713699-14748-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.103.61:44111
--webserver_interface=127.14.103.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.61.keytab
--principal=kudu/127.14.103.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.103.20:33271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.103.62:35903,127.14.103.61:44111,127.14.103.60:44099
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786501898713699-14748-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786501898713699-14748-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786501898713699-14748-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 02:34:46.623792 29900 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.103.61" port: 44111 } attempt: 1
I20260812 02:34:46.624766 29903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:46.624461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.61:44111 (local address 127.0.0.1:57350)
0812 02:34:46.624622 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0812 02:34:46.624688 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.61:44111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260812 02:34:46.625586 29903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:46.625348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.61:44111 (local address 127.0.0.1:57356)
0812 02:34:46.625474 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0812 02:34:46.625531 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.61:44111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260812 02:34:46.625725 29900 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.103.61:44111: Network error: Client connection negotiation failed: client connection to 127.14.103.61:44111: connect: Connection refused (error 111)
I20260812 02:34:46.693909 29900 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.103.61" port: 44111 } attempt: 2
I20260812 02:34:46.694865 29903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:46.694544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.61:44111 (local address 127.0.0.1:57360)
0812 02:34:46.694715 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0812 02:34:46.694788 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.61:44111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260812 02:34:46.695703 29903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:46.695437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.61:44111 (local address 127.0.0.1:57364)
0812 02:34:46.695588 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0812 02:34:46.695642 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.61:44111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
W20260812 02:34:46.695816 29900 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.103.61:44111: Network error: Client connection negotiation failed: client connection to 127.14.103.61:44111: connect: Connection refused (error 111)
W20260812 02:34:46.704236 29907 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:34:46.704602 29907 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:34:46.704691 29907 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:34:46.709084 29907 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:34:46.709190 29907 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:34:46.709230 29907 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:34:46.709285 29907 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:34:46.709336 29907 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 02:34:46.715808 29907 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.103.20:33271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786501898713699-14748-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786501898713699-14748-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.14.103.62:35903,127.14.103.61:44111,127.14.103.60:44099
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.61.keytab
--principal=kudu/127.14.103.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.103.61:44111
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786501898713699-14748-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.14.103.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29907
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786501898713699-14748-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:34:46.717048 29907 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:34:46.718130 29907 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:34:46.727828 29914 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:34:46.727828 29912 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:34:46.727907 29916 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:34:46.728281 29907 server_base.cc:1034] running on GCE node
I20260812 02:34:46.729036 29907 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:34:46.730338 29907 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:34:46.731549 29907 hybrid_clock.cc:648] HybridClock initialized: now 1786502086731513 us; error 56 us; skew 500 ppm
Aug 12 02:34:46 dist-test-slave-2w3w krb5kdc[29807](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502086, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:34:46.734869 29907 init.cc:377] Logged in from keytab as kudu/127.14.103.61@KRBTEST.COM (short username kudu)
I20260812 02:34:46.735909 29907 webserver.cc:460] Webserver started at http://127.14.103.61:40137/ using document root <none> and password file <none>
I20260812 02:34:46.736462 29907 fs_manager.cc:359] Metadata directory not provided
I20260812 02:34:46.736527 29907 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:34:46.736701 29907 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:34:46.738804 29907 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786501898713699-14748-0/minicluster-data/master-1/data/instance:
uuid: "27ef8369729b4fd0bbdb7e3eb882246d"
format_stamp: "Formatted at 2026-08-12 02:34:46 on dist-test-slave-2w3w"
server_key: "98270abe5ae91112a4576d8f11c7f287"
server_key_iv: "652ee61e3d22b3bfaf0f2be8587f20d2"
server_key_version: "encryptionkey@0"
I20260812 02:34:46.739508 29907 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786501898713699-14748-0/minicluster-data/master-1/wal/instance:
uuid: "27ef8369729b4fd0bbdb7e3eb882246d"
format_stamp: "Formatted at 2026-08-12 02:34:46 on dist-test-slave-2w3w"
server_key: "98270abe5ae91112a4576d8f11c7f287"
server_key_iv: "652ee61e3d22b3bfaf0f2be8587f20d2"
server_key_version: "encryptionkey@0"
I20260812 02:34:46.743569 29907 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260812 02:34:46.746232 29923 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:34:46.747632 29907 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 02:34:46.747750 29907 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786501898713699-14748-0/minicluster-data/master-1/data,/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786501898713699-14748-0/minicluster-data/master-1/wal
uuid: "27ef8369729b4fd0bbdb7e3eb882246d"
format_stamp: "Formatted at 2026-08-12 02:34:46 on dist-test-slave-2w3w"
server_key: "98270abe5ae91112a4576d8f11c7f287"
server_key_iv: "652ee61e3d22b3bfaf0f2be8587f20d2"
server_key_version: "encryptionkey@0"
I20260812 02:34:46.747859 29907 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786501898713699-14748-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786501898713699-14748-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786501898713699-14748-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:34:46.766554 29907 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:34:46.770340 29907 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:34:46.770524 29907 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:34:46.778280 29907 rpc_server.cc:307] RPC server started. Bound to: 127.14.103.61:44111
I20260812 02:34:46.778311 29980 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.103.61:44111 every 8 connection(s)
I20260812 02:34:46.779412 29907 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786501898713699-14748-0/minicluster-data/master-1/data/info.pb
I20260812 02:34:46.780272 14748 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbep4M4/build/debug/bin/kudu as pid 29907
I20260812 02:34:46.780382 14748 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786501898713699-14748-0/minicluster-data/master-1/wal/instance
I20260812 02:34:46.780659 14748 external_mini_cluster.cc:1428] Setting key b20d209470c33b388e7d47a53bedd8ad
I20260812 02:34:46.782788 29981 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:34:46.787046 29900 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.103.61" port: 44111 } attempt: 3
I20260812 02:34:46.789144 29981 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.103.62" port: 35903 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.14.103.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.103.60@KRBTEST.COM" created.
Aug 12 02:34:46 dist-test-slave-2w3w krb5kdc[29807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502086, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.62@KRBTEST.COM for kudu/127.14.103.61@KRBTEST.COM
Aug 12 02:34:46 dist-test-slave-2w3w krb5kdc[29807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502086, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.61@KRBTEST.COM for kudu/127.14.103.62@KRBTEST.COM
I20260812 02:34:46.803869 29903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:46.787657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.61:44111 (local address 127.0.0.1:57366)
0812 02:34:46.787897 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0812 02:34:46.787915 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 02:34:46.788016 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:34:46.790899 (+ 2883us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:34:46.790902 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:34:46.791189 (+ 287us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:34:46.791343 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:34:46.791352 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:34:46.793269 (+ 1917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:34:46.793275 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:34:46.793833 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:34:46.793841 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:34:46.794885 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:34:46.794889 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:34:46.795086 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:46.795728 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:34:46.795744 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:34:46.797635 (+ 1891us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:34:46.800905 (+ 3270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:34:46.800911 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:34:46.800915 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:34:46.801219 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:34:46.801676 (+ 457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:34:46.801680 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:34:46.801682 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:34:46.801806 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:34:46.803349 (+ 1543us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:34:46.803354 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:34:46.803483 (+ 129us) client_negotiation.cc:769] Sending connection context
0812 02:34:46.803557 (+ 74us) client_negotiation.cc:240] Negotiation successful
0812 02:34:46.803678 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"mutex_wait_us":1,"spinlock_wait_cycles":6144}
I20260812 02:34:46.805512 29904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:46.792132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42166 (local address 127.14.103.62:35903)
0812 02:34:46.792525 (+ 393us) server_negotiation.cc:206] Beginning negotiation
0812 02:34:46.792530 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:34:46.792831 (+ 301us) server_negotiation.cc:407] Connection header received
0812 02:34:46.793043 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:34:46.793046 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:34:46.793097 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:34:46.793321 (+ 224us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:34:46.794565 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:46.795469 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:46.796953 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:46.797383 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:46.797545 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:46.800531 (+ 2986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:34:46.800538 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:34:46.800544 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:34:46.800584 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:34:46.802382 (+ 1798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:34:46.802931 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:34:46.802935 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:34:46.802937 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:34:46.803073 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:34:46.803405 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:34:46.803410 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:34:46.803412 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:34:46.803677 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:34:46.803799 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0812 02:34:46.805238 (+ 1439us) server_negotiation.cc:299] Negotiation successful
0812 02:34:46.805370 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":33,"server-negotiator.queue_time_us":57}
I20260812 02:34:46.805954 29985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:46.792075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:35903 (local address 127.0.0.1:42166)
0812 02:34:46.792705 (+ 630us) negotiation.cc:107] Waiting for socket to connect
0812 02:34:46.792747 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0812 02:34:46.792908 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:34:46.793384 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:34:46.793390 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:34:46.793928 (+ 538us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:34:46.794415 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:34:46.794431 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:34:46.795630 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:34:46.795632 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:34:46.796807 (+ 1175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:34:46.796818 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:34:46.797611 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:34:46.797615 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:34:46.797760 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:46.798349 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:34:46.798369 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:34:46.800341 (+ 1972us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:34:46.802501 (+ 2160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:34:46.802515 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:34:46.802522 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:34:46.802810 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:34:46.803185 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:34:46.803189 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:34:46.803191 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:34:46.803290 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:34:46.803808 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:34:46.803814 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:34:46.804140 (+ 326us) client_negotiation.cc:769] Sending connection context
0812 02:34:46.804336 (+ 196us) client_negotiation.cc:240] Negotiation successful
0812 02:34:46.804804 (+ 468us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":468,"mutex_wait_us":150,"thread_start_us":279,"threads_started":1}
I20260812 02:34:46.805958 29984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:46.788479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57366 (local address 127.14.103.61:44111)
0812 02:34:46.788984 (+ 505us) server_negotiation.cc:206] Beginning negotiation
0812 02:34:46.789000 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0812 02:34:46.789039 (+ 39us) server_negotiation.cc:407] Connection header received
0812 02:34:46.790089 (+ 1050us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:34:46.790112 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:34:46.790574 (+ 462us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:34:46.790949 (+ 375us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:34:46.791485 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:46.793124 (+ 1639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:46.793972 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:46.794676 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:46.794939 (+ 263us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:46.797798 (+ 2859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:34:46.797806 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:34:46.797823 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:34:46.797862 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:34:46.800340 (+ 2478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:34:46.801356 (+ 1016us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:34:46.801365 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:34:46.801373 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:34:46.801535 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:34:46.802529 (+ 994us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:34:46.802533 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:34:46.802534 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:34:46.803211 (+ 677us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:34:46.803927 (+ 716us) server_negotiation.cc:1035] Waiting for connection context
0812 02:34:46.804355 (+ 428us) server_negotiation.cc:299] Negotiation successful
0812 02:34:46.804716 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":48,"server-negotiator.queue_time_us":343,"spinlock_wait_cycles":896,"thread_start_us":185,"threads_started":1}
I20260812 02:34:46.807392 29981 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.103.61" port: 44111 } has no permanent_uuid. Determining permanent_uuid...
I20260812 02:34:46.808276 29900 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.103.60" port: 44099 } has no permanent_uuid. Determining permanent_uuid...
I20260812 02:34:46.808983 29903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:46.808727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.60:44099 (local address 127.0.0.1:43060)
0812 02:34:46.808842 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0812 02:34:46.808900 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.60:44099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260812 02:34:46.809708 29903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:46.809384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.60:44099 (local address 127.0.0.1:43070)
0812 02:34:46.809558 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0812 02:34:46.809641 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.60:44099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
W20260812 02:34:46.809916 29900 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.103.60:44099: Network error: Client connection negotiation failed: client connection to 127.14.103.60:44099: connect: Connection refused (error 111)
Aug 12 02:34:46 dist-test-slave-2w3w krb5kdc[29807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502086, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.61@KRBTEST.COM for kudu/127.14.103.61@KRBTEST.COM
I20260812 02:34:46.816512 29984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:46.807994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57376 (local address 127.14.103.61:44111)
0812 02:34:46.808114 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0812 02:34:46.808118 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:34:46.808133 (+ 15us) server_negotiation.cc:407] Connection header received
0812 02:34:46.808273 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:34:46.808276 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:34:46.808325 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:34:46.808438 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:34:46.808932 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:46.809548 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:46.810437 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:46.810669 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:46.810775 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:46.813062 (+ 2287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:34:46.813066 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:34:46.813068 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:34:46.813091 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:34:46.814470 (+ 1379us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:34:46.815150 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:34:46.815154 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:34:46.815156 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:34:46.815249 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:34:46.815534 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:34:46.815536 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:34:46.815538 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:34:46.815794 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:34:46.815901 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0812 02:34:46.816156 (+ 255us) server_negotiation.cc:299] Negotiation successful
0812 02:34:46.816387 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":116,"server-negotiator.queue_time_us":40}
I20260812 02:34:46.816512 29985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:46.807905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.61:44111 (local address 127.0.0.1:57376)
0812 02:34:46.808047 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0812 02:34:46.808062 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 02:34:46.808178 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:34:46.808438 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:34:46.808441 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:34:46.808647 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:34:46.808789 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:34:46.808800 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:34:46.809662 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:34:46.809668 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:34:46.810319 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:34:46.810329 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:34:46.810776 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:34:46.810779 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:34:46.810876 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:46.811436 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:34:46.811451 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:34:46.812908 (+ 1457us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:34:46.814597 (+ 1689us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:34:46.814603 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:34:46.814605 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:34:46.814820 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:34:46.815349 (+ 529us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:34:46.815352 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:34:46.815354 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:34:46.815440 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:34:46.815901 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:34:46.815904 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:34:46.815995 (+ 91us) client_negotiation.cc:769] Sending connection context
0812 02:34:46.816100 (+ 105us) client_negotiation.cc:240] Negotiation successful
0812 02:34:46.816315 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.103.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.60.keytab.
Entry for principal kudu/127.14.103.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.60.keytab.
I20260812 02:34:46.817562 29981 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.103.60" port: 44099 } has no permanent_uuid. Determining permanent_uuid...
I20260812 02:34:46.818218 14748 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbep4M4/build/debug/bin/kudu
/tmp/dist-test-taskbep4M4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786501898713699-14748-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786501898713699-14748-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786501898713699-14748-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786501898713699-14748-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.103.60:44099
--webserver_interface=127.14.103.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.60.keytab
--principal=kudu/127.14.103.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.103.20:33271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.103.62:35903,127.14.103.61:44111,127.14.103.60:44099
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786501898713699-14748-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786501898713699-14748-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786501898713699-14748-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 02:34:46.818403 29985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:46.818055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.60:44099 (local address 127.0.0.1:43082)
0812 02:34:46.818176 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0812 02:34:46.818303 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.60:44099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260812 02:34:46.819221 29924 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.103.60:44099: connect: Connection refused (error 111)
I20260812 02:34:46.820485 29985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:46.820072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.60:44099 (local address 127.0.0.1:43092)
0812 02:34:46.820349 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0812 02:34:46.820415 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.60:44099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202}
W20260812 02:34:46.820750 29981 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.103.60:44099: Network error: Client connection negotiation failed: client connection to 127.14.103.60:44099: connect: Connection refused (error 111)
I20260812 02:34:46.841351 29900 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.103.60" port: 44099 } attempt: 1
I20260812 02:34:46.842729 29903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:46.842109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.60:44099 (local address 127.0.0.1:43094)
0812 02:34:46.842423 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0812 02:34:46.842583 (+ 160us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.60:44099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138}
I20260812 02:34:46.844328 29903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:46.843929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.60:44099 (local address 127.0.0.1:43102)
0812 02:34:46.844128 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0812 02:34:46.844209 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.60:44099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
W20260812 02:34:46.844524 29900 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.103.60:44099: Network error: Client connection negotiation failed: client connection to 127.14.103.60:44099: connect: Connection refused (error 111)
I20260812 02:34:46.870083 29981 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.103.60" port: 44099 } attempt: 1
I20260812 02:34:46.871279 29985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:46.870778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.60:44099 (local address 127.0.0.1:43112)
0812 02:34:46.871042 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0812 02:34:46.871187 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.60:44099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260812 02:34:46.872467 29985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:46.872165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.60:44099 (local address 127.0.0.1:43118)
0812 02:34:46.872323 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0812 02:34:46.872387 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.60:44099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
W20260812 02:34:46.872592 29981 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.103.60:44099: Network error: Client connection negotiation failed: client connection to 127.14.103.60:44099: connect: Connection refused (error 111)
I20260812 02:34:46.919721 29900 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.103.60" port: 44099 } attempt: 2
I20260812 02:34:46.920639 29903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:46.920329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.60:44099 (local address 127.0.0.1:43120)
0812 02:34:46.920491 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0812 02:34:46.920559 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.60:44099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260812 02:34:46.921492 29903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:46.921271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.60:44099 (local address 127.0.0.1:43124)
0812 02:34:46.921382 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0812 02:34:46.921435 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.60:44099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260812 02:34:46.921721 29900 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.103.60:44099: Network error: Client connection negotiation failed: client connection to 127.14.103.60:44099: connect: Connection refused (error 111)
I20260812 02:34:46.940821 29981 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.103.60" port: 44099 } attempt: 2
I20260812 02:34:46.942430 29985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:46.942028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.60:44099 (local address 127.0.0.1:43134)
0812 02:34:46.942260 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0812 02:34:46.942333 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.60:44099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115}
I20260812 02:34:46.943450 29985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:46.943177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.60:44099 (local address 127.0.0.1:43146)
0812 02:34:46.943319 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0812 02:34:46.943384 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.60:44099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260812 02:34:46.943692 29981 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.103.60:44099: Network error: Client connection negotiation failed: client connection to 127.14.103.60:44099: connect: Connection refused (error 111)
W20260812 02:34:46.948446 29988 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:34:46.948766 29988 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:34:46.948920 29988 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:34:46.953640 29988 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:34:46.953735 29988 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:34:46.953755 29988 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:34:46.953773 29988 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:34:46.953790 29988 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 02:34:46.959815 29988 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.103.20:33271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786501898713699-14748-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786501898713699-14748-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.14.103.62:35903,127.14.103.61:44111,127.14.103.60:44099
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.60.keytab
--principal=kudu/127.14.103.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.103.60:44099
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786501898713699-14748-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.14.103.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29988
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786501898713699-14748-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:34:46.960994 29988 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:34:46.962049 29988 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:34:46.971832 29995 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:34:46.971832 29993 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:34:46.971932 29997 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:34:46.972424 29988 server_base.cc:1034] running on GCE node
I20260812 02:34:46.973106 29988 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:34:46.974324 29988 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:34:46.975505 29988 hybrid_clock.cc:648] HybridClock initialized: now 1786502086975476 us; error 49 us; skew 500 ppm
Aug 12 02:34:46 dist-test-slave-2w3w krb5kdc[29807](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502086, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:34:46.979440 29988 init.cc:377] Logged in from keytab as kudu/127.14.103.60@KRBTEST.COM (short username kudu)
I20260812 02:34:46.981109 29988 webserver.cc:460] Webserver started at http://127.14.103.60:37039/ using document root <none> and password file <none>
I20260812 02:34:46.981968 29988 fs_manager.cc:359] Metadata directory not provided
I20260812 02:34:46.982064 29988 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:34:46.982508 29988 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:34:46.985165 29988 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786501898713699-14748-0/minicluster-data/master-2/data/instance:
uuid: "2622b62c81074f4bb96937905be2d93d"
format_stamp: "Formatted at 2026-08-12 02:34:46 on dist-test-slave-2w3w"
server_key: "871274da7b52391e937258bd2825aa87"
server_key_iv: "dddeedaef3d79975d7f98477121600e6"
server_key_version: "encryptionkey@0"
I20260812 02:34:46.985814 29988 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786501898713699-14748-0/minicluster-data/master-2/wal/instance:
uuid: "2622b62c81074f4bb96937905be2d93d"
format_stamp: "Formatted at 2026-08-12 02:34:46 on dist-test-slave-2w3w"
server_key: "871274da7b52391e937258bd2825aa87"
server_key_iv: "dddeedaef3d79975d7f98477121600e6"
server_key_version: "encryptionkey@0"
I20260812 02:34:46.990285 29988 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260812 02:34:46.993911 30005 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:34:46.996347 29988 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 02:34:46.996640 29988 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786501898713699-14748-0/minicluster-data/master-2/data,/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786501898713699-14748-0/minicluster-data/master-2/wal
uuid: "2622b62c81074f4bb96937905be2d93d"
format_stamp: "Formatted at 2026-08-12 02:34:46 on dist-test-slave-2w3w"
server_key: "871274da7b52391e937258bd2825aa87"
server_key_iv: "dddeedaef3d79975d7f98477121600e6"
server_key_version: "encryptionkey@0"
I20260812 02:34:46.996837 29988 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786501898713699-14748-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786501898713699-14748-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786501898713699-14748-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:34:47.013394 29988 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:34:47.017795 29988 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:34:47.018031 29988 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:34:47.020903 29900 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.103.60" port: 44099 } attempt: 3
I20260812 02:34:47.021955 29903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:47.021550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.60:44099 (local address 127.0.0.1:43162)
0812 02:34:47.021736 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0812 02:34:47.021813 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.60:44099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260812 02:34:47.022867 29903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:47.022586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.60:44099 (local address 127.0.0.1:43164)
0812 02:34:47.022727 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0812 02:34:47.022795 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.60:44099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260812 02:34:47.023070 29900 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.103.60:44099: Network error: Client connection negotiation failed: client connection to 127.14.103.60:44099: connect: Connection refused (error 111)
I20260812 02:34:47.026775 29988 rpc_server.cc:307] RPC server started. Bound to: 127.14.103.60:44099
I20260812 02:34:47.026801 30062 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.103.60:44099 every 8 connection(s)
I20260812 02:34:47.027866 29988 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786501898713699-14748-0/minicluster-data/master-2/data/info.pb
I20260812 02:34:47.030908 30063 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:34:47.034914 29981 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.103.60" port: 44099 } attempt: 3
I20260812 02:34:47.035490 30063 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.103.62" port: 35903 } has no permanent_uuid. Determining permanent_uuid...
I20260812 02:34:47.037886 14748 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbep4M4/build/debug/bin/kudu as pid 29988
I20260812 02:34:47.038043 14748 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786501898713699-14748-0/minicluster-data/master-2/wal/instance
I20260812 02:34:47.038386 14748 external_mini_cluster.cc:1428] Setting key ad385ef051781334b9587297020f80ad
Aug 12 02:34:47 dist-test-slave-2w3w krb5kdc[29807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502086, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.61@KRBTEST.COM for kudu/127.14.103.60@KRBTEST.COM
Aug 12 02:34:47 dist-test-slave-2w3w krb5kdc[29807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502086, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.60@KRBTEST.COM for kudu/127.14.103.62@KRBTEST.COM
Aug 12 02:34:47 dist-test-slave-2w3w krb5kdc[29807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502086, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.103.60@KRBTEST.COM
I20260812 02:34:47.053587 29904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:47.037614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42172 (local address 127.14.103.62:35903)
0812 02:34:47.037755 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0812 02:34:47.037760 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:34:47.038237 (+ 477us) server_negotiation.cc:407] Connection header received
0812 02:34:47.038675 (+ 438us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:34:47.038681 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:34:47.038740 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:34:47.038825 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:34:47.040199 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:47.040951 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:47.042397 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:47.042642 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:47.042763 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:47.047666 (+ 4903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:34:47.047674 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:34:47.047679 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:34:47.047721 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:34:47.050521 (+ 2800us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:34:47.051342 (+ 821us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:34:47.051347 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:34:47.051349 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:34:47.051467 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:34:47.051997 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:34:47.052002 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:34:47.052004 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:34:47.052317 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:34:47.052415 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0812 02:34:47.053202 (+ 787us) server_negotiation.cc:299] Negotiation successful
0812 02:34:47.053394 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260812 02:34:47.053685 29985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:47.035580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.60:44099 (local address 127.0.0.1:43176)
0812 02:34:47.035780 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0812 02:34:47.035802 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0812 02:34:47.035918 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:34:47.038736 (+ 2818us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:34:47.038740 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:34:47.039020 (+ 280us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:34:47.039141 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:34:47.039149 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:34:47.041126 (+ 1977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:34:47.041130 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:34:47.041973 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:34:47.041987 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:34:47.042900 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:34:47.042904 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:34:47.043042 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:47.043805 (+ 763us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:34:47.043835 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:34:47.046062 (+ 2227us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:34:47.049687 (+ 3625us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:34:47.049691 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:34:47.049694 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:34:47.051579 (+ 1885us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:34:47.052076 (+ 497us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:34:47.052080 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:34:47.052082 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:34:47.052193 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:34:47.053115 (+ 922us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:34:47.053118 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:34:47.053235 (+ 117us) client_negotiation.cc:769] Sending connection context
0812 02:34:47.053328 (+ 93us) client_negotiation.cc:240] Negotiation successful
0812 02:34:47.053502 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67}
I20260812 02:34:47.055632 30066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:47.037511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:35903 (local address 127.0.0.1:42172)
0812 02:34:47.038077 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0812 02:34:47.038137 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0812 02:34:47.038437 (+ 300us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:34:47.039051 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:34:47.039061 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:34:47.039537 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:34:47.040034 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:34:47.040048 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:34:47.041081 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:34:47.041084 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:34:47.042270 (+ 1186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:34:47.042279 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:34:47.042822 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:34:47.042827 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:34:47.043034 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:47.043906 (+ 872us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:34:47.043932 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:34:47.047407 (+ 3475us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:34:47.050696 (+ 3289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:34:47.050708 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:34:47.050723 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:34:47.051160 (+ 437us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:34:47.051615 (+ 455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:34:47.051618 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:34:47.051620 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:34:47.051866 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:34:47.052502 (+ 636us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:34:47.052509 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:34:47.052917 (+ 408us) client_negotiation.cc:769] Sending connection context
0812 02:34:47.054208 (+ 1291us) client_negotiation.cc:240] Negotiation successful
0812 02:34:47.054574 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":365,"mutex_wait_us":114,"thread_start_us":87,"threads_started":1}
I20260812 02:34:47.055629 30067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:47.039083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43190 (local address 127.14.103.60:44099)
0812 02:34:47.039725 (+ 642us) server_negotiation.cc:206] Beginning negotiation
0812 02:34:47.039732 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 02:34:47.039743 (+ 11us) server_negotiation.cc:407] Connection header received
0812 02:34:47.039774 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:34:47.039776 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:34:47.039809 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:34:47.039910 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:34:47.040801 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:47.041734 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:47.042876 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:47.043264 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:47.043384 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:47.048298 (+ 4914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:34:47.048304 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:34:47.048308 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:34:47.048342 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:34:47.051577 (+ 3235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:34:47.052192 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:34:47.052195 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:34:47.052197 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:34:47.052320 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:34:47.052730 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:34:47.052733 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:34:47.052735 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:34:47.052968 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:34:47.053194 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0812 02:34:47.053955 (+ 761us) server_negotiation.cc:299] Negotiation successful
0812 02:34:47.054455 (+ 500us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":156,"server-negotiator.queue_time_us":586,"thread_start_us":109,"threads_started":1}
I20260812 02:34:47.055629 30065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:47.036406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43176 (local address 127.14.103.60:44099)
0812 02:34:47.036870 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0812 02:34:47.036885 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0812 02:34:47.036940 (+ 55us) server_negotiation.cc:407] Connection header received
0812 02:34:47.037900 (+ 960us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:34:47.037924 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:34:47.038456 (+ 532us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:34:47.039331 (+ 875us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:34:47.039810 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:47.040966 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:47.042123 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:47.042716 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:47.043343 (+ 627us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:47.046428 (+ 3085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:34:47.046439 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:34:47.046490 (+ 51us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:34:47.046526 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:34:47.049470 (+ 2944us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:34:47.051751 (+ 2281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:34:47.051758 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:34:47.051759 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:34:47.051868 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:34:47.052321 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:34:47.052325 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:34:47.052326 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:34:47.052995 (+ 669us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:34:47.053975 (+ 980us) server_negotiation.cc:1035] Waiting for connection context
0812 02:34:47.054039 (+ 64us) server_negotiation.cc:299] Negotiation successful
0812 02:34:47.054380 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":30,"server-negotiator.queue_time_us":297,"spinlock_wait_cycles":4352,"thread_start_us":155,"threads_started":1}
I20260812 02:34:47.058421 30063 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.103.61" port: 44111 } has no permanent_uuid. Determining permanent_uuid...
I20260812 02:34:47.062179 29981 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 27ef8369729b4fd0bbdb7e3eb882246d: Bootstrap starting.
I20260812 02:34:47.065272 29981 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 27ef8369729b4fd0bbdb7e3eb882246d: Neither blocks nor log segments found. Creating new log.
Aug 12 02:34:47 dist-test-slave-2w3w krb5kdc[29807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502086, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.103.62@KRBTEST.COM
I20260812 02:34:47.066309 29981 log.cc:824] T 00000000000000000000000000000000 P 27ef8369729b4fd0bbdb7e3eb882246d: Log is configured to *not* fsync() on all Append() calls
I20260812 02:34:47.069060 29981 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 27ef8369729b4fd0bbdb7e3eb882246d: No bootstrap required, opened a new log
I20260812 02:34:47.071485 29904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:47.060054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42184 (local address 127.14.103.62:35903)
0812 02:34:47.060190 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0812 02:34:47.060194 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:34:47.060208 (+ 14us) server_negotiation.cc:407] Connection header received
0812 02:34:47.060307 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:34:47.060311 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:34:47.060360 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:34:47.060447 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:34:47.061399 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:47.061925 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:47.062814 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:47.063132 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:47.063239 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:47.066208 (+ 2969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:34:47.066212 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:34:47.066214 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:34:47.066245 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:34:47.069378 (+ 3133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:34:47.069960 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:34:47.069964 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:34:47.069966 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:34:47.070074 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:34:47.070431 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:34:47.070437 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:34:47.070442 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:34:47.070706 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:34:47.070986 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0812 02:34:47.071137 (+ 151us) server_negotiation.cc:299] Negotiation successful
0812 02:34:47.071277 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
Aug 12 02:34:47 dist-test-slave-2w3w krb5kdc[29807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502086, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.60@KRBTEST.COM for kudu/127.14.103.61@KRBTEST.COM
I20260812 02:34:47.074702 29981 raft_consensus.cc:348] T 00000000000000000000000000000000 P 27ef8369729b4fd0bbdb7e3eb882246d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "915d272f1ca84bbbb64f7a710f4482f2" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 35903 } } peers { permanent_uuid: "27ef8369729b4fd0bbdb7e3eb882246d" member_type: VOTER last_known_addr { host: "127.14.103.61" port: 44111 } } peers { permanent_uuid: "2622b62c81074f4bb96937905be2d93d" member_type: VOTER last_known_addr { host: "127.14.103.60" port: 44099 } }
I20260812 02:34:47.075219 29981 raft_consensus.cc:374] T 00000000000000000000000000000000 P 27ef8369729b4fd0bbdb7e3eb882246d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:34:47.075304 29981 raft_consensus.cc:729] T 00000000000000000000000000000000 P 27ef8369729b4fd0bbdb7e3eb882246d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27ef8369729b4fd0bbdb7e3eb882246d, State: Initialized, Role: FOLLOWER
I20260812 02:34:47.076149 29981 consensus_queue.cc:260] T 00000000000000000000000000000000 P 27ef8369729b4fd0bbdb7e3eb882246d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "915d272f1ca84bbbb64f7a710f4482f2" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 35903 } } peers { permanent_uuid: "27ef8369729b4fd0bbdb7e3eb882246d" member_type: VOTER last_known_addr { host: "127.14.103.61" port: 44111 } } peers { permanent_uuid: "2622b62c81074f4bb96937905be2d93d" member_type: VOTER last_known_addr { host: "127.14.103.60" port: 44099 } }
I20260812 02:34:47.076929 30066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:47.059269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.61:44111 (local address 127.0.0.1:57386)
0812 02:34:47.059403 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0812 02:34:47.059418 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 02:34:47.059515 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:34:47.067518 (+ 8003us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:34:47.067522 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:34:47.067738 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:34:47.067932 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:34:47.067941 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:34:47.068718 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:34:47.068722 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:34:47.069523 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:34:47.069538 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:34:47.070143 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:34:47.070147 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:34:47.070264 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:47.070799 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:34:47.070815 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:34:47.072599 (+ 1784us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:34:47.075192 (+ 2593us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:34:47.075196 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:34:47.075198 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:34:47.075507 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:34:47.075825 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:34:47.075828 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:34:47.075830 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:34:47.075922 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:34:47.076358 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:34:47.076361 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:34:47.076473 (+ 112us) client_negotiation.cc:769] Sending connection context
0812 02:34:47.076583 (+ 110us) client_negotiation.cc:240] Negotiation successful
0812 02:34:47.076753 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260812 02:34:47.076929 29984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:47.066940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57386 (local address 127.14.103.61:44111)
0812 02:34:47.067154 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0812 02:34:47.067161 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 02:34:47.067181 (+ 20us) server_negotiation.cc:407] Connection header received
0812 02:34:47.067233 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:34:47.067236 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:34:47.067284 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:34:47.067443 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:34:47.068080 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:47.068590 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:47.069663 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:47.070018 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:47.070134 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:47.072785 (+ 2651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:34:47.072789 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:34:47.072792 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:34:47.072822 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:34:47.075048 (+ 2226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:34:47.075615 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:34:47.075620 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:34:47.075622 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:34:47.075716 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:34:47.076014 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:34:47.076018 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:34:47.076020 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:34:47.076257 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:34:47.076369 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0812 02:34:47.076637 (+ 268us) server_negotiation.cc:299] Negotiation successful
0812 02:34:47.076771 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110}
I20260812 02:34:47.077741 30063 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.103.60" port: 44099 } has no permanent_uuid. Determining permanent_uuid...
I20260812 02:34:47.078387 29981 sys_catalog.cc:564] T 00000000000000000000000000000000 P 27ef8369729b4fd0bbdb7e3eb882246d [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:34:47.079248 30071 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27ef8369729b4fd0bbdb7e3eb882246d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "915d272f1ca84bbbb64f7a710f4482f2" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 35903 } } peers { permanent_uuid: "27ef8369729b4fd0bbdb7e3eb882246d" member_type: VOTER last_known_addr { host: "127.14.103.61" port: 44111 } } peers { permanent_uuid: "2622b62c81074f4bb96937905be2d93d" member_type: VOTER last_known_addr { host: "127.14.103.60" port: 44099 } } }
I20260812 02:34:47.079550 30071 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27ef8369729b4fd0bbdb7e3eb882246d [sys.catalog]: This master's current role is: FOLLOWER
Aug 12 02:34:47 dist-test-slave-2w3w krb5kdc[29807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502086, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.103.61@KRBTEST.COM
W20260812 02:34:47.085569 30082 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 27ef8369729b4fd0bbdb7e3eb882246d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 02:34:47.085683 30082 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
Aug 12 02:34:47 dist-test-slave-2w3w krb5kdc[29807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502086, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.60@KRBTEST.COM for kudu/127.14.103.60@KRBTEST.COM
I20260812 02:34:47.087666 30070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:47.073074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57390 (local address 127.14.103.61:44111)
0812 02:34:47.073549 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0812 02:34:47.073558 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 02:34:47.073572 (+ 14us) server_negotiation.cc:407] Connection header received
0812 02:34:47.073611 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:34:47.073614 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:34:47.073658 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:34:47.073759 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:34:47.075272 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:47.076651 (+ 1379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:47.078579 (+ 1928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:47.078992 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:47.079100 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:47.083168 (+ 4068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:34:47.083172 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:34:47.083175 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:34:47.083203 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:34:47.085734 (+ 2531us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:34:47.086276 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:34:47.086281 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:34:47.086282 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:34:47.086379 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:34:47.086674 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:34:47.086678 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:34:47.086680 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:34:47.086900 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:34:47.087153 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0812 02:34:47.087314 (+ 161us) server_negotiation.cc:299] Negotiation successful
0812 02:34:47.087445 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":185,"threads_started":1}
I20260812 02:34:47.083748 29907 master_runner.cc:186] Error getting master registration for 127.14.103.62:35903: OK, instance_id { permanent_uuid: "915d272f1ca84bbbb64f7a710f4482f2" instance_seqno: 1786502086538362 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 02:34:47.088227 29907 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260812 02:34:47.091776 14748 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 02:34:47.092880 30066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:47.078296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.60:44099 (local address 127.0.0.1:43196)
0812 02:34:47.078397 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0812 02:34:47.078412 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 02:34:47.078496 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:34:47.078774 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:34:47.078777 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:34:47.079024 (+ 247us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:34:47.079129 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:34:47.079137 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:34:47.080339 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:34:47.080342 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:34:47.081220 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:34:47.081231 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:34:47.081942 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:34:47.081945 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:34:47.082018 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:47.082953 (+ 935us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:34:47.083012 (+ 59us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:34:47.088073 (+ 5061us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:34:47.090347 (+ 2274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:34:47.090351 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:34:47.090353 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:34:47.090873 (+ 520us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:34:47.091287 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:34:47.091290 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:34:47.091292 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:34:47.091492 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:34:47.092259 (+ 767us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:34:47.092262 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:34:47.092393 (+ 131us) client_negotiation.cc:769] Sending connection context
0812 02:34:47.092509 (+ 116us) client_negotiation.cc:240] Negotiation successful
0812 02:34:47.092692 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260812 02:34:47.092885 30067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:47.078486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43196 (local address 127.14.103.60:44099)
0812 02:34:47.078592 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0812 02:34:47.078595 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 02:34:47.078606 (+ 11us) server_negotiation.cc:407] Connection header received
0812 02:34:47.078640 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:34:47.078642 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:34:47.078674 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:34:47.078755 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:34:47.079253 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:47.080180 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:47.081369 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:47.081695 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:47.081775 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:47.088257 (+ 6482us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:34:47.088261 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:34:47.088264 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:34:47.088292 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:34:47.090210 (+ 1918us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:34:47.091065 (+ 855us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:34:47.091071 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:34:47.091076 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:34:47.091181 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:34:47.091706 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:34:47.091713 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:34:47.091714 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:34:47.092067 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:34:47.092273 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0812 02:34:47.092574 (+ 301us) server_negotiation.cc:299] Negotiation successful
0812 02:34:47.092758 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":72,"server-negotiator.queue_time_us":44}
I20260812 02:34:47.096648 30063 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2622b62c81074f4bb96937905be2d93d: Bootstrap starting.
I20260812 02:34:47.099901 30063 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2622b62c81074f4bb96937905be2d93d: Neither blocks nor log segments found. Creating new log.
I20260812 02:34:47.101006 30063 log.cc:824] T 00000000000000000000000000000000 P 2622b62c81074f4bb96937905be2d93d: Log is configured to *not* fsync() on all Append() calls
I20260812 02:34:47.111971 29904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:47.099153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42200 (local address 127.14.103.62:35903)
0812 02:34:47.099307 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0812 02:34:47.099312 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:34:47.099332 (+ 20us) server_negotiation.cc:407] Connection header received
0812 02:34:47.099388 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:34:47.099390 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:34:47.099441 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:34:47.099541 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:34:47.100790 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:47.101542 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:47.103094 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:47.103449 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:47.103561 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:47.106090 (+ 2529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:34:47.106096 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:34:47.106099 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:34:47.106141 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:34:47.109184 (+ 3043us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:34:47.109985 (+ 801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:34:47.109989 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:34:47.109991 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:34:47.110093 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:34:47.110418 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:34:47.110421 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:34:47.110423 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:34:47.110702 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:34:47.110811 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0812 02:34:47.111568 (+ 757us) server_negotiation.cc:299] Negotiation successful
0812 02:34:47.111699 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260812 02:34:47.112461 30067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:47.096097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43206 (local address 127.14.103.60:44099)
0812 02:34:47.099329 (+ 3232us) server_negotiation.cc:206] Beginning negotiation
0812 02:34:47.099334 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:34:47.099350 (+ 16us) server_negotiation.cc:407] Connection header received
0812 02:34:47.099387 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:34:47.099390 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:34:47.099441 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:34:47.099543 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:34:47.102053 (+ 2510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:47.103012 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:47.104083 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:47.104456 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:47.104571 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:47.106561 (+ 1990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:34:47.106567 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:34:47.106573 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:34:47.106618 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:34:47.108537 (+ 1919us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:34:47.109123 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:34:47.109128 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:34:47.109130 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:34:47.109230 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:34:47.109585 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:34:47.109588 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:34:47.109592 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:34:47.109877 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:34:47.110000 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0812 02:34:47.111867 (+ 1867us) server_negotiation.cc:299] Negotiation successful
0812 02:34:47.112005 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3149}
I20260812 02:34:47.112746 30070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:47.095708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57404 (local address 127.14.103.61:44111)
0812 02:34:47.101586 (+ 5878us) server_negotiation.cc:206] Beginning negotiation
0812 02:34:47.101593 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 02:34:47.101614 (+ 21us) server_negotiation.cc:407] Connection header received
0812 02:34:47.102074 (+ 460us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:34:47.102079 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:34:47.102135 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:34:47.102212 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:34:47.103246 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:47.104190 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:47.105756 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:47.106211 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:47.106340 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:47.108481 (+ 2141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:34:47.108486 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:34:47.108489 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:34:47.108530 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:34:47.110324 (+ 1794us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:34:47.110840 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:34:47.110844 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:34:47.110846 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:34:47.110935 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:34:47.111272 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:34:47.111275 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:34:47.111277 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:34:47.111517 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:34:47.111648 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0812 02:34:47.112307 (+ 659us) server_negotiation.cc:299] Negotiation successful
0812 02:34:47.112452 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5761}
I20260812 02:34:47.114526 30063 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2622b62c81074f4bb96937905be2d93d: No bootstrap required, opened a new log
I20260812 02:34:47.118732 30063 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2622b62c81074f4bb96937905be2d93d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "915d272f1ca84bbbb64f7a710f4482f2" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 35903 } } peers { permanent_uuid: "27ef8369729b4fd0bbdb7e3eb882246d" member_type: VOTER last_known_addr { host: "127.14.103.61" port: 44111 } } peers { permanent_uuid: "2622b62c81074f4bb96937905be2d93d" member_type: VOTER last_known_addr { host: "127.14.103.60" port: 44099 } }
I20260812 02:34:47.119117 30063 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2622b62c81074f4bb96937905be2d93d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:34:47.119185 30063 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2622b62c81074f4bb96937905be2d93d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2622b62c81074f4bb96937905be2d93d, State: Initialized, Role: FOLLOWER
I20260812 02:34:47.119743 30063 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2622b62c81074f4bb96937905be2d93d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "915d272f1ca84bbbb64f7a710f4482f2" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 35903 } } peers { permanent_uuid: "27ef8369729b4fd0bbdb7e3eb882246d" member_type: VOTER last_known_addr { host: "127.14.103.61" port: 44111 } } peers { permanent_uuid: "2622b62c81074f4bb96937905be2d93d" member_type: VOTER last_known_addr { host: "127.14.103.60" port: 44099 } }
I20260812 02:34:47.120420 30092 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2622b62c81074f4bb96937905be2d93d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "915d272f1ca84bbbb64f7a710f4482f2" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 35903 } } peers { permanent_uuid: "27ef8369729b4fd0bbdb7e3eb882246d" member_type: VOTER last_known_addr { host: "127.14.103.61" port: 44111 } } peers { permanent_uuid: "2622b62c81074f4bb96937905be2d93d" member_type: VOTER last_known_addr { host: "127.14.103.60" port: 44099 } } }
I20260812 02:34:47.120596 30092 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2622b62c81074f4bb96937905be2d93d [sys.catalog]: This master's current role is: FOLLOWER
I20260812 02:34:47.121167 30063 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2622b62c81074f4bb96937905be2d93d [sys.catalog]: configured and running, proceeding with master startup.
W20260812 02:34:47.127293 30105 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2622b62c81074f4bb96937905be2d93d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 02:34:47.127399 30105 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 02:34:47.125545 29988 master_runner.cc:186] Error getting master registration for 127.14.103.62:35903: OK, instance_id { permanent_uuid: "915d272f1ca84bbbb64f7a710f4482f2" instance_seqno: 1786502086538362 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 02:34:47.128909 29988 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260812 02:34:47.187260 29900 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.103.60" port: 44099 } attempt: 4
Aug 12 02:34:47 dist-test-slave-2w3w krb5kdc[29807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502086, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.62@KRBTEST.COM for kudu/127.14.103.60@KRBTEST.COM
I20260812 02:34:47.199841 29903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:47.187992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.60:44099 (local address 127.0.0.1:43220)
0812 02:34:47.188221 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0812 02:34:47.188237 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 02:34:47.188348 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:34:47.188655 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:34:47.188658 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:34:47.188899 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:34:47.189020 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:34:47.189028 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:34:47.190101 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:34:47.190105 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:34:47.191034 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:34:47.191046 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:34:47.191674 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:34:47.191678 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:34:47.191774 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:47.192516 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:34:47.192533 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:34:47.195013 (+ 2480us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:34:47.197800 (+ 2787us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:34:47.197805 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:34:47.197808 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:34:47.198164 (+ 356us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:34:47.198511 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:34:47.198514 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:34:47.198516 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:34:47.198619 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:34:47.199224 (+ 605us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:34:47.199227 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:34:47.199356 (+ 129us) client_negotiation.cc:769] Sending connection context
0812 02:34:47.199469 (+ 113us) client_negotiation.cc:240] Negotiation successful
0812 02:34:47.199648 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138}
I20260812 02:34:47.199873 30067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:47.188095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43220 (local address 127.14.103.60:44099)
0812 02:34:47.188229 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0812 02:34:47.188234 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:34:47.188320 (+ 86us) server_negotiation.cc:407] Connection header received
0812 02:34:47.188455 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:34:47.188458 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:34:47.188512 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:34:47.188618 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:34:47.189160 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:47.189963 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:47.191180 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:47.191539 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:47.191657 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:47.195313 (+ 3656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:34:47.195319 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:34:47.195322 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:34:47.195362 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:34:47.197604 (+ 2242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:34:47.198291 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:34:47.198296 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:34:47.198298 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:34:47.198405 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:34:47.198775 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:34:47.198779 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:34:47.198781 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:34:47.199086 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:34:47.199205 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0812 02:34:47.199566 (+ 361us) server_negotiation.cc:299] Negotiation successful
0812 02:34:47.199708 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260812 02:34:47.204190 29900 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 915d272f1ca84bbbb64f7a710f4482f2: Bootstrap starting.
I20260812 02:34:47.207542 29900 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 915d272f1ca84bbbb64f7a710f4482f2: Neither blocks nor log segments found. Creating new log.
I20260812 02:34:47.210127 29900 log.cc:824] T 00000000000000000000000000000000 P 915d272f1ca84bbbb64f7a710f4482f2: Log is configured to *not* fsync() on all Append() calls
I20260812 02:34:47.213479 29900 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 915d272f1ca84bbbb64f7a710f4482f2: No bootstrap required, opened a new log
I20260812 02:34:47.217993 29900 raft_consensus.cc:348] T 00000000000000000000000000000000 P 915d272f1ca84bbbb64f7a710f4482f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "915d272f1ca84bbbb64f7a710f4482f2" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 35903 } } peers { permanent_uuid: "27ef8369729b4fd0bbdb7e3eb882246d" member_type: VOTER last_known_addr { host: "127.14.103.61" port: 44111 } } peers { permanent_uuid: "2622b62c81074f4bb96937905be2d93d" member_type: VOTER last_known_addr { host: "127.14.103.60" port: 44099 } }
I20260812 02:34:47.218349 29900 raft_consensus.cc:374] T 00000000000000000000000000000000 P 915d272f1ca84bbbb64f7a710f4482f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:34:47.218427 29900 raft_consensus.cc:729] T 00000000000000000000000000000000 P 915d272f1ca84bbbb64f7a710f4482f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 915d272f1ca84bbbb64f7a710f4482f2, State: Initialized, Role: FOLLOWER
I20260812 02:34:47.218953 29900 consensus_queue.cc:260] T 00000000000000000000000000000000 P 915d272f1ca84bbbb64f7a710f4482f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "915d272f1ca84bbbb64f7a710f4482f2" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 35903 } } peers { permanent_uuid: "27ef8369729b4fd0bbdb7e3eb882246d" member_type: VOTER last_known_addr { host: "127.14.103.61" port: 44111 } } peers { permanent_uuid: "2622b62c81074f4bb96937905be2d93d" member_type: VOTER last_known_addr { host: "127.14.103.60" port: 44099 } }
I20260812 02:34:47.219755 30109 sys_catalog.cc:455] T 00000000000000000000000000000000 P 915d272f1ca84bbbb64f7a710f4482f2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "915d272f1ca84bbbb64f7a710f4482f2" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 35903 } } peers { permanent_uuid: "27ef8369729b4fd0bbdb7e3eb882246d" member_type: VOTER last_known_addr { host: "127.14.103.61" port: 44111 } } peers { permanent_uuid: "2622b62c81074f4bb96937905be2d93d" member_type: VOTER last_known_addr { host: "127.14.103.60" port: 44099 } } }
I20260812 02:34:47.219988 30109 sys_catalog.cc:458] T 00000000000000000000000000000000 P 915d272f1ca84bbbb64f7a710f4482f2 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 02:34:47.220696 29900 sys_catalog.cc:564] T 00000000000000000000000000000000 P 915d272f1ca84bbbb64f7a710f4482f2 [sys.catalog]: configured and running, proceeding with master startup.
W20260812 02:34:47.227770 30122 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 915d272f1ca84bbbb64f7a710f4482f2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 02:34:47.227974 30122 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 02:34:47.230607 29822 master_runner.cc:263] No leader master found from master 915d272f1ca84bbbb64f7a710f4482f2
I20260812 02:34:47.230705 29822 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260812 02:34:47.269850 30013 tablet.cc:2369] T 00000000000000000000000000000000 P 2622b62c81074f4bb96937905be2d93d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 02:34:47.272289 29929 tablet.cc:2369] T 00000000000000000000000000000000 P 27ef8369729b4fd0bbdb7e3eb882246d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 02:34:47.284102 29848 tablet.cc:2369] T 00000000000000000000000000000000 P 915d272f1ca84bbbb64f7a710f4482f2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 02:34:47.318676 30092 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2622b62c81074f4bb96937905be2d93d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 02:34:47.318984 30092 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2622b62c81074f4bb96937905be2d93d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "915d272f1ca84bbbb64f7a710f4482f2" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 35903 } } peers { permanent_uuid: "27ef8369729b4fd0bbdb7e3eb882246d" member_type: VOTER last_known_addr { host: "127.14.103.61" port: 44111 } } peers { permanent_uuid: "2622b62c81074f4bb96937905be2d93d" member_type: VOTER last_known_addr { host: "127.14.103.60" port: 44099 } }
I20260812 02:34:47.320317 30092 leader_election.cc:288] T 00000000000000000000000000000000 P 2622b62c81074f4bb96937905be2d93d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 915d272f1ca84bbbb64f7a710f4482f2 (127.14.103.62:35903), 27ef8369729b4fd0bbdb7e3eb882246d (127.14.103.61:44111)
I20260812 02:34:47.321044 29872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2622b62c81074f4bb96937905be2d93d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "915d272f1ca84bbbb64f7a710f4482f2" is_pre_election: true
I20260812 02:34:47.321035 29949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2622b62c81074f4bb96937905be2d93d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27ef8369729b4fd0bbdb7e3eb882246d" is_pre_election: true
I20260812 02:34:47.321437 29949 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 27ef8369729b4fd0bbdb7e3eb882246d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2622b62c81074f4bb96937905be2d93d in term 0.
I20260812 02:34:47.321437 29872 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 915d272f1ca84bbbb64f7a710f4482f2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2622b62c81074f4bb96937905be2d93d in term 0.
I20260812 02:34:47.322022 30006 leader_election.cc:302] T 00000000000000000000000000000000 P 2622b62c81074f4bb96937905be2d93d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2622b62c81074f4bb96937905be2d93d, 27ef8369729b4fd0bbdb7e3eb882246d, 915d272f1ca84bbbb64f7a710f4482f2; no voters:
I20260812 02:34:47.322386 30092 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2622b62c81074f4bb96937905be2d93d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 02:34:47.322486 30092 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2622b62c81074f4bb96937905be2d93d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 02:34:47.322551 30092 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2622b62c81074f4bb96937905be2d93d [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:34:47.324429 30092 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2622b62c81074f4bb96937905be2d93d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "915d272f1ca84bbbb64f7a710f4482f2" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 35903 } } peers { permanent_uuid: "27ef8369729b4fd0bbdb7e3eb882246d" member_type: VOTER last_known_addr { host: "127.14.103.61" port: 44111 } } peers { permanent_uuid: "2622b62c81074f4bb96937905be2d93d" member_type: VOTER last_known_addr { host: "127.14.103.60" port: 44099 } }
I20260812 02:34:47.324900 30092 leader_election.cc:288] T 00000000000000000000000000000000 P 2622b62c81074f4bb96937905be2d93d [CANDIDATE]: Term 1 election: Requested vote from peers 915d272f1ca84bbbb64f7a710f4482f2 (127.14.103.62:35903), 27ef8369729b4fd0bbdb7e3eb882246d (127.14.103.61:44111)
I20260812 02:34:47.325438 29949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2622b62c81074f4bb96937905be2d93d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27ef8369729b4fd0bbdb7e3eb882246d"
I20260812 02:34:47.325435 29872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2622b62c81074f4bb96937905be2d93d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "915d272f1ca84bbbb64f7a710f4482f2"
I20260812 02:34:47.325600 29949 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 27ef8369729b4fd0bbdb7e3eb882246d [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:34:47.325600 29872 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 915d272f1ca84bbbb64f7a710f4482f2 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:34:47.327191 29872 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 915d272f1ca84bbbb64f7a710f4482f2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2622b62c81074f4bb96937905be2d93d in term 1.
I20260812 02:34:47.327453 29949 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 27ef8369729b4fd0bbdb7e3eb882246d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2622b62c81074f4bb96937905be2d93d in term 1.
I20260812 02:34:47.327575 30006 leader_election.cc:302] T 00000000000000000000000000000000 P 2622b62c81074f4bb96937905be2d93d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2622b62c81074f4bb96937905be2d93d, 915d272f1ca84bbbb64f7a710f4482f2; no voters:
I20260812 02:34:47.327819 30092 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2622b62c81074f4bb96937905be2d93d [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:34:47.328096 30092 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2622b62c81074f4bb96937905be2d93d [term 1 LEADER]: Becoming Leader. State: Replica: 2622b62c81074f4bb96937905be2d93d, State: Running, Role: LEADER
I20260812 02:34:47.328567 30092 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2622b62c81074f4bb96937905be2d93d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "915d272f1ca84bbbb64f7a710f4482f2" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 35903 } } peers { permanent_uuid: "27ef8369729b4fd0bbdb7e3eb882246d" member_type: VOTER last_known_addr { host: "127.14.103.61" port: 44111 } } peers { permanent_uuid: "2622b62c81074f4bb96937905be2d93d" member_type: VOTER last_known_addr { host: "127.14.103.60" port: 44099 } }
I20260812 02:34:47.330904 30125 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2622b62c81074f4bb96937905be2d93d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2622b62c81074f4bb96937905be2d93d. Latest consensus state: current_term: 1 leader_uuid: "2622b62c81074f4bb96937905be2d93d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "915d272f1ca84bbbb64f7a710f4482f2" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 35903 } } peers { permanent_uuid: "27ef8369729b4fd0bbdb7e3eb882246d" member_type: VOTER last_known_addr { host: "127.14.103.61" port: 44111 } } peers { permanent_uuid: "2622b62c81074f4bb96937905be2d93d" member_type: VOTER last_known_addr { host: "127.14.103.60" port: 44099 } } }
I20260812 02:34:47.331101 30125 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2622b62c81074f4bb96937905be2d93d [sys.catalog]: This master's current role is: LEADER
I20260812 02:34:47.331560 30127 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 02:34:47.332525 30127 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 02:34:47.337826 29949 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 27ef8369729b4fd0bbdb7e3eb882246d [term 1 FOLLOWER]: Refusing update from remote peer 2622b62c81074f4bb96937905be2d93d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:34:47.337801 29872 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 915d272f1ca84bbbb64f7a710f4482f2 [term 1 FOLLOWER]: Refusing update from remote peer 2622b62c81074f4bb96937905be2d93d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:34:47.338605 30092 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2622b62c81074f4bb96937905be2d93d [LEADER]: Connected to new peer: Peer: permanent_uuid: "915d272f1ca84bbbb64f7a710f4482f2" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 35903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:34:47.338891 30125 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2622b62c81074f4bb96937905be2d93d [LEADER]: Connected to new peer: Peer: permanent_uuid: "27ef8369729b4fd0bbdb7e3eb882246d" member_type: VOTER last_known_addr { host: "127.14.103.61" port: 44111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:34:47.343926 30109 sys_catalog.cc:455] T 00000000000000000000000000000000 P 915d272f1ca84bbbb64f7a710f4482f2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2622b62c81074f4bb96937905be2d93d. Latest consensus state: current_term: 1 leader_uuid: "2622b62c81074f4bb96937905be2d93d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "915d272f1ca84bbbb64f7a710f4482f2" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 35903 } } peers { permanent_uuid: "27ef8369729b4fd0bbdb7e3eb882246d" member_type: VOTER last_known_addr { host: "127.14.103.61" port: 44111 } } peers { permanent_uuid: "2622b62c81074f4bb96937905be2d93d" member_type: VOTER last_known_addr { host: "127.14.103.60" port: 44099 } } }
I20260812 02:34:47.343926 30071 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27ef8369729b4fd0bbdb7e3eb882246d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2622b62c81074f4bb96937905be2d93d. Latest consensus state: current_term: 1 leader_uuid: "2622b62c81074f4bb96937905be2d93d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "915d272f1ca84bbbb64f7a710f4482f2" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 35903 } } peers { permanent_uuid: "27ef8369729b4fd0bbdb7e3eb882246d" member_type: VOTER last_known_addr { host: "127.14.103.61" port: 44111 } } peers { permanent_uuid: "2622b62c81074f4bb96937905be2d93d" member_type: VOTER last_known_addr { host: "127.14.103.60" port: 44099 } } }
I20260812 02:34:47.344101 30071 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27ef8369729b4fd0bbdb7e3eb882246d [sys.catalog]: This master's current role is: FOLLOWER
I20260812 02:34:47.344101 30109 sys_catalog.cc:458] T 00000000000000000000000000000000 P 915d272f1ca84bbbb64f7a710f4482f2 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 02:34:47.346988 30125 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2622b62c81074f4bb96937905be2d93d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2622b62c81074f4bb96937905be2d93d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "915d272f1ca84bbbb64f7a710f4482f2" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 35903 } } peers { permanent_uuid: "27ef8369729b4fd0bbdb7e3eb882246d" member_type: VOTER last_known_addr { host: "127.14.103.61" port: 44111 } } peers { permanent_uuid: "2622b62c81074f4bb96937905be2d93d" member_type: VOTER last_known_addr { host: "127.14.103.60" port: 44099 } } }
I20260812 02:34:47.347162 30125 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2622b62c81074f4bb96937905be2d93d [sys.catalog]: This master's current role is: LEADER
I20260812 02:34:47.347554 30125 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2622b62c81074f4bb96937905be2d93d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2622b62c81074f4bb96937905be2d93d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "915d272f1ca84bbbb64f7a710f4482f2" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 35903 } } peers { permanent_uuid: "27ef8369729b4fd0bbdb7e3eb882246d" member_type: VOTER last_known_addr { host: "127.14.103.61" port: 44111 } } peers { permanent_uuid: "2622b62c81074f4bb96937905be2d93d" member_type: VOTER last_known_addr { host: "127.14.103.60" port: 44099 } } }
I20260812 02:34:47.347661 30125 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2622b62c81074f4bb96937905be2d93d [sys.catalog]: This master's current role is: LEADER
I20260812 02:34:47.348385 30071 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27ef8369729b4fd0bbdb7e3eb882246d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2622b62c81074f4bb96937905be2d93d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "915d272f1ca84bbbb64f7a710f4482f2" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 35903 } } peers { permanent_uuid: "27ef8369729b4fd0bbdb7e3eb882246d" member_type: VOTER last_known_addr { host: "127.14.103.61" port: 44111 } } peers { permanent_uuid: "2622b62c81074f4bb96937905be2d93d" member_type: VOTER last_known_addr { host: "127.14.103.60" port: 44099 } } }
I20260812 02:34:47.348523 30071 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27ef8369729b4fd0bbdb7e3eb882246d [sys.catalog]: This master's current role is: FOLLOWER
I20260812 02:34:47.348659 30128 mvcc.cc:205] Tried to move back new op lower bound from 7317512549727539200 to 7317512549701111808. Current Snapshot: MvccSnapshot[applied={T|T < 7317512549727539200}]
I20260812 02:34:47.348873 30109 sys_catalog.cc:455] T 00000000000000000000000000000000 P 915d272f1ca84bbbb64f7a710f4482f2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2622b62c81074f4bb96937905be2d93d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "915d272f1ca84bbbb64f7a710f4482f2" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 35903 } } peers { permanent_uuid: "27ef8369729b4fd0bbdb7e3eb882246d" member_type: VOTER last_known_addr { host: "127.14.103.61" port: 44111 } } peers { permanent_uuid: "2622b62c81074f4bb96937905be2d93d" member_type: VOTER last_known_addr { host: "127.14.103.60" port: 44099 } } }
I20260812 02:34:47.348981 30109 sys_catalog.cc:458] T 00000000000000000000000000000000 P 915d272f1ca84bbbb64f7a710f4482f2 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 02:34:47.349396 30130 mvcc.cc:205] Tried to move back new op lower bound from 7317512549727539200 to 7317512549701111808. Current Snapshot: MvccSnapshot[applied={T|T < 7317512549727539200}]
I20260812 02:34:47.352047 30127 catalog_manager.cc:1347] Generated new cluster ID: 90aa580284b948dea5024b6a95a740d2
I20260812 02:34:47.352154 30127 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 02:34:47.361759 30127 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 02:34:47.363153 30127 catalog_manager.cc:1504] Loading token signing keys...
I20260812 02:34:47.379195 30127 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2622b62c81074f4bb96937905be2d93d: Generated new TSK 0
I20260812 02:34:47.380074 30127 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 02:34:48.087119 30082 catalog_manager.cc:1259] Loaded cluster ID: 90aa580284b948dea5024b6a95a740d2
I20260812 02:34:48.087208 30082 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 27ef8369729b4fd0bbdb7e3eb882246d: loading cluster ID for follower catalog manager: success
I20260812 02:34:48.088950 30082 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 27ef8369729b4fd0bbdb7e3eb882246d: acquiring CA information for follower catalog manager: success
I20260812 02:34:48.090286 30082 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 27ef8369729b4fd0bbdb7e3eb882246d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 02:34:48.229436 30122 catalog_manager.cc:1259] Loaded cluster ID: 90aa580284b948dea5024b6a95a740d2
I20260812 02:34:48.229535 30122 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 915d272f1ca84bbbb64f7a710f4482f2: loading cluster ID for follower catalog manager: success
I20260812 02:34:48.231369 30122 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 915d272f1ca84bbbb64f7a710f4482f2: acquiring CA information for follower catalog manager: success
I20260812 02:34:48.232662 30122 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 915d272f1ca84bbbb64f7a710f4482f2: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 02:34:48.452878 14748 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbep4M4/build/debug/bin/kudu with pid 29822
I20260812 02:34:48.462157 14748 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbep4M4/build/debug/bin/kudu with pid 29907
I20260812 02:34:48.471825 14748 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbep4M4/build/debug/bin/kudu with pid 29988
2026-08-12T02:34:48Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3428 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786501898713699-14748-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 02:34:48 dist-test-slave-2w3w krb5kdc[30144](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 02:34:48 dist-test-slave-2w3w krb5kdc[30144](info): set up 2 sockets
Aug 12 02:34:48 dist-test-slave-2w3w krb5kdc[30144](info): commencing operation
krb5kdc: starting...
W20260812 02:34:49.638742 14748 mini_kdc.cc:121] Time spent starting KDC: real 1.141s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786501898713699-14748-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786501898713699-14748-0/krb5kdc/test-user.keytab.
Aug 12 02:34:49 dist-test-slave-2w3w krb5kdc[30144](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502089, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T02:34:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T02:34:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.103.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.103.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.103.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.62.keytab.
Entry for principal kudu/127.14.103.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.62.keytab.
I20260812 02:34:49.799314 14748 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbep4M4/build/debug/bin/kudu
/tmp/dist-test-taskbep4M4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786501898713699-14748-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786501898713699-14748-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786501898713699-14748-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786501898713699-14748-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.103.62:35617
--webserver_interface=127.14.103.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.62.keytab
--principal=kudu/127.14.103.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.103.20:41883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.103.62:35617
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786501898713699-14748-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786501898713699-14748-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786501898713699-14748-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:34:49.920533 30158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:34:49.921059 30158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:34:49.921185 30158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:34:49.926066 30158 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:34:49.926221 30158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:34:49.926265 30158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:34:49.926306 30158 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:34:49.926359 30158 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 02:34:49.932907 30158 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.103.20:41883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786501898713699-14748-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786501898713699-14748-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.103.62:35617
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.62.keytab
--principal=kudu/127.14.103.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.103.62:35617
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786501898713699-14748-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.103.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30158
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786501898713699-14748-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:34:49.934386 30158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:34:49.935581 30158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:34:49.945538 30167 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:34:49.945746 30158 server_base.cc:1034] running on GCE node
W20260812 02:34:49.945559 30163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:34:49.945582 30164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:34:49.946681 30158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:34:49.948004 30158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:34:49.949237 30158 hybrid_clock.cc:648] HybridClock initialized: now 1786502089949199 us; error 55 us; skew 500 ppm
Aug 12 02:34:49 dist-test-slave-2w3w krb5kdc[30144](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502089, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:34:49.952577 30158 init.cc:377] Logged in from keytab as kudu/127.14.103.62@KRBTEST.COM (short username kudu)
I20260812 02:34:49.953712 30158 webserver.cc:460] Webserver started at http://127.14.103.62:38015/ using document root <none> and password file <none>
I20260812 02:34:49.954352 30158 fs_manager.cc:359] Metadata directory not provided
I20260812 02:34:49.954459 30158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:34:49.954708 30158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:34:49.957056 30158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786501898713699-14748-0/minicluster-data/master-0/data/instance:
uuid: "a1dcbaf1575b4da88f1beab43d55742f"
format_stamp: "Formatted at 2026-08-12 02:34:49 on dist-test-slave-2w3w"
server_key: "f46693006d012120cc2b5a7057670364"
server_key_iv: "e9d78ad89a0c8472e2212600a761801f"
server_key_version: "encryptionkey@0"
I20260812 02:34:49.957695 30158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786501898713699-14748-0/minicluster-data/master-0/wal/instance:
uuid: "a1dcbaf1575b4da88f1beab43d55742f"
format_stamp: "Formatted at 2026-08-12 02:34:49 on dist-test-slave-2w3w"
server_key: "f46693006d012120cc2b5a7057670364"
server_key_iv: "e9d78ad89a0c8472e2212600a761801f"
server_key_version: "encryptionkey@0"
I20260812 02:34:49.962088 30158 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260812 02:34:49.965240 30174 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:34:49.966884 30158 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260812 02:34:49.967118 30158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786501898713699-14748-0/minicluster-data/master-0/data,/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786501898713699-14748-0/minicluster-data/master-0/wal
uuid: "a1dcbaf1575b4da88f1beab43d55742f"
format_stamp: "Formatted at 2026-08-12 02:34:49 on dist-test-slave-2w3w"
server_key: "f46693006d012120cc2b5a7057670364"
server_key_iv: "e9d78ad89a0c8472e2212600a761801f"
server_key_version: "encryptionkey@0"
I20260812 02:34:49.967273 30158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786501898713699-14748-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786501898713699-14748-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786501898713699-14748-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:34:49.993173 30158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:34:49.997202 30158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:34:49.997459 30158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:34:50.005740 30158 rpc_server.cc:307] RPC server started. Bound to: 127.14.103.62:35617
I20260812 02:34:50.005756 30233 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.103.62:35617 every 8 connection(s)
I20260812 02:34:50.006837 30158 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786501898713699-14748-0/minicluster-data/master-0/data/info.pb
I20260812 02:34:50.008196 14748 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbep4M4/build/debug/bin/kudu as pid 30158
I20260812 02:34:50.008342 14748 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786501898713699-14748-0/minicluster-data/master-0/wal/instance
I20260812 02:34:50.008653 14748 external_mini_cluster.cc:1428] Setting key de4cb92a472b0b0ae601705a7d4d294e
I20260812 02:34:50.010624 30234 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:34:50.016813 30234 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1dcbaf1575b4da88f1beab43d55742f: Bootstrap starting.
Aug 12 02:34:50 dist-test-slave-2w3w krb5kdc[30144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502089, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.103.62@KRBTEST.COM
I20260812 02:34:50.020051 30234 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a1dcbaf1575b4da88f1beab43d55742f: Neither blocks nor log segments found. Creating new log.
I20260812 02:34:50.021015 30234 log.cc:824] T 00000000000000000000000000000000 P a1dcbaf1575b4da88f1beab43d55742f: Log is configured to *not* fsync() on all Append() calls
I20260812 02:34:50.023237 30234 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1dcbaf1575b4da88f1beab43d55742f: No bootstrap required, opened a new log
I20260812 02:34:50.024951 30237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:50.010311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33600 (local address 127.14.103.62:35617)
0812 02:34:50.010876 (+ 565us) server_negotiation.cc:206] Beginning negotiation
0812 02:34:50.010893 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0812 02:34:50.010932 (+ 39us) server_negotiation.cc:407] Connection header received
0812 02:34:50.011702 (+ 770us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:34:50.011719 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:34:50.012073 (+ 354us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:34:50.012458 (+ 385us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:34:50.013340 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:50.014385 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:50.015386 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:50.015906 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:50.016085 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:50.019545 (+ 3460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:34:50.019552 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:34:50.019569 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:34:50.019607 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:34:50.021746 (+ 2139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:34:50.022317 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:34:50.022323 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:34:50.022328 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:34:50.022459 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:34:50.022749 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:34:50.022753 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:34:50.022754 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:34:50.023249 (+ 495us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:34:50.023471 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0812 02:34:50.023760 (+ 289us) server_negotiation.cc:299] Negotiation successful
0812 02:34:50.024067 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":156,"threads_started":1}
I20260812 02:34:50.027333 30234 raft_consensus.cc:348] T 00000000000000000000000000000000 P a1dcbaf1575b4da88f1beab43d55742f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1dcbaf1575b4da88f1beab43d55742f" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 35617 } }
I20260812 02:34:50.027650 30234 raft_consensus.cc:374] T 00000000000000000000000000000000 P a1dcbaf1575b4da88f1beab43d55742f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:34:50.027747 30234 raft_consensus.cc:729] T 00000000000000000000000000000000 P a1dcbaf1575b4da88f1beab43d55742f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1dcbaf1575b4da88f1beab43d55742f, State: Initialized, Role: FOLLOWER
I20260812 02:34:50.028216 14748 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 02:34:50.028235 30234 consensus_queue.cc:260] T 00000000000000000000000000000000 P a1dcbaf1575b4da88f1beab43d55742f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1dcbaf1575b4da88f1beab43d55742f" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 35617 } }
I20260812 02:34:50.028406 30234 raft_consensus.cc:388] T 00000000000000000000000000000000 P a1dcbaf1575b4da88f1beab43d55742f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 02:34:50.028489 30234 raft_consensus.cc:482] T 00000000000000000000000000000000 P a1dcbaf1575b4da88f1beab43d55742f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 02:34:50.028589 30234 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a1dcbaf1575b4da88f1beab43d55742f [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:34:50.029632 30234 raft_consensus.cc:504] T 00000000000000000000000000000000 P a1dcbaf1575b4da88f1beab43d55742f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1dcbaf1575b4da88f1beab43d55742f" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 35617 } }
I20260812 02:34:50.030006 30234 leader_election.cc:302] T 00000000000000000000000000000000 P a1dcbaf1575b4da88f1beab43d55742f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a1dcbaf1575b4da88f1beab43d55742f; no voters:
I20260812 02:34:50.030342 30234 leader_election.cc:288] T 00000000000000000000000000000000 P a1dcbaf1575b4da88f1beab43d55742f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 02:34:50.030505 30239 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a1dcbaf1575b4da88f1beab43d55742f [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:34:50.030757 30239 raft_consensus.cc:686] T 00000000000000000000000000000000 P a1dcbaf1575b4da88f1beab43d55742f [term 1 LEADER]: Becoming Leader. State: Replica: a1dcbaf1575b4da88f1beab43d55742f, State: Running, Role: LEADER
I20260812 02:34:50.031136 30239 consensus_queue.cc:237] T 00000000000000000000000000000000 P a1dcbaf1575b4da88f1beab43d55742f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1dcbaf1575b4da88f1beab43d55742f" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 35617 } }
I20260812 02:34:50.031682 30234 sys_catalog.cc:564] T 00000000000000000000000000000000 P a1dcbaf1575b4da88f1beab43d55742f [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:34:50.032788 30243 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1dcbaf1575b4da88f1beab43d55742f [sys.catalog]: SysCatalogTable state changed. Reason: New leader a1dcbaf1575b4da88f1beab43d55742f. Latest consensus state: current_term: 1 leader_uuid: "a1dcbaf1575b4da88f1beab43d55742f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1dcbaf1575b4da88f1beab43d55742f" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 35617 } } }
I20260812 02:34:50.032920 30243 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1dcbaf1575b4da88f1beab43d55742f [sys.catalog]: This master's current role is: LEADER
I20260812 02:34:50.033336 30250 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 02:34:50.033293 30241 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1dcbaf1575b4da88f1beab43d55742f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a1dcbaf1575b4da88f1beab43d55742f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1dcbaf1575b4da88f1beab43d55742f" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 35617 } } }
I20260812 02:34:50.033465 30241 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1dcbaf1575b4da88f1beab43d55742f [sys.catalog]: This master's current role is: LEADER
I20260812 02:34:50.036650 30250 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 02:34:50.043007 30250 catalog_manager.cc:1347] Generated new cluster ID: c32d401a44e34133a41b18f3246273ae
I20260812 02:34:50.043107 30250 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 02:34:50.054373 30250 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 02:34:50.055474 30250 catalog_manager.cc:1504] Loading token signing keys...
I20260812 02:34:50.067330 30250 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a1dcbaf1575b4da88f1beab43d55742f: Generated new TSK 0
I20260812 02:34:50.068104 30250 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 02:34:50.084815 14748 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbep4M4/build/debug/bin/kudu with pid 30158
2026-08-12T02:34:50Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1610 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 02:34:50 dist-test-slave-2w3w krb5kdc[30263](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 02:34:50 dist-test-slave-2w3w krb5kdc[30263](info): set up 2 sockets
Aug 12 02:34:50 dist-test-slave-2w3w krb5kdc[30263](info): commencing operation
krb5kdc: starting...
W20260812 02:34:51.235633 14748 mini_kdc.cc:121] Time spent starting KDC: real 1.127s user 0.000s sys 0.009s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/krb5kdc/test-user.keytab.
Aug 12 02:34:51 dist-test-slave-2w3w krb5kdc[30263](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502091, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T02:34:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T02:34:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.103.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.103.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.103.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.62.keytab.
Entry for principal kudu/127.14.103.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.62.keytab.
I20260812 02:34:51.396800 14748 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbep4M4/build/debug/bin/kudu
/tmp/dist-test-taskbep4M4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.103.62:43307
--webserver_interface=127.14.103.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.62.keytab
--principal=kudu/127.14.103.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.103.20:36423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.103.62:43307
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:34:51.521960 30277 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:34:51.522332 30277 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:34:51.522409 30277 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:34:51.526715 30277 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:34:51.526844 30277 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:34:51.526886 30277 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:34:51.526927 30277 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:34:51.526986 30277 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 02:34:51.533463 30277 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.103.20:36423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.103.62:43307
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.62.keytab
--principal=kudu/127.14.103.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.103.62:43307
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.103.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30277
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:34:51.534933 30277 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:34:51.536170 30277 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:34:51.545833 30282 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:34:51.545785 30283 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:34:51.545982 30277 server_base.cc:1034] running on GCE node
W20260812 02:34:51.545823 30285 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:34:51.546900 30277 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:34:51.548293 30277 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:34:51.549479 30277 hybrid_clock.cc:648] HybridClock initialized: now 1786502091549465 us; error 33 us; skew 500 ppm
Aug 12 02:34:51 dist-test-slave-2w3w krb5kdc[30263](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502091, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:34:51.552620 30277 init.cc:377] Logged in from keytab as kudu/127.14.103.62@KRBTEST.COM (short username kudu)
I20260812 02:34:51.553586 30277 webserver.cc:460] Webserver started at http://127.14.103.62:36911/ using document root <none> and password file <none>
I20260812 02:34:51.554136 30277 fs_manager.cc:359] Metadata directory not provided
I20260812 02:34:51.554200 30277 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:34:51.554385 30277 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:34:51.556609 30277 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/minicluster-data/master-0/data/instance:
uuid: "d92eab9524324b82b329fd272df49540"
format_stamp: "Formatted at 2026-08-12 02:34:51 on dist-test-slave-2w3w"
server_key: "664a157ae4b40297525e3039b779c47b"
server_key_iv: "295b86a832b5ed934d9001353bd656fc"
server_key_version: "encryptionkey@0"
I20260812 02:34:51.557173 30277 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/minicluster-data/master-0/wal/instance:
uuid: "d92eab9524324b82b329fd272df49540"
format_stamp: "Formatted at 2026-08-12 02:34:51 on dist-test-slave-2w3w"
server_key: "664a157ae4b40297525e3039b779c47b"
server_key_iv: "295b86a832b5ed934d9001353bd656fc"
server_key_version: "encryptionkey@0"
I20260812 02:34:51.561226 30277 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260812 02:34:51.563853 30294 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:34:51.565200 30277 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260812 02:34:51.565371 30277 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/minicluster-data/master-0/data,/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/minicluster-data/master-0/wal
uuid: "d92eab9524324b82b329fd272df49540"
format_stamp: "Formatted at 2026-08-12 02:34:51 on dist-test-slave-2w3w"
server_key: "664a157ae4b40297525e3039b779c47b"
server_key_iv: "295b86a832b5ed934d9001353bd656fc"
server_key_version: "encryptionkey@0"
I20260812 02:34:51.565514 30277 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:34:51.588196 30277 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:34:51.591925 30277 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:34:51.592168 30277 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:34:51.599938 30277 rpc_server.cc:307] RPC server started. Bound to: 127.14.103.62:43307
I20260812 02:34:51.600289 30350 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.103.62:43307 every 8 connection(s)
I20260812 02:34:51.601038 30277 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/minicluster-data/master-0/data/info.pb
I20260812 02:34:51.604135 30351 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:34:51.606319 14748 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbep4M4/build/debug/bin/kudu as pid 30277
I20260812 02:34:51.606467 14748 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/minicluster-data/master-0/wal/instance
I20260812 02:34:51.606782 14748 external_mini_cluster.cc:1428] Setting key 4c603f50ce9e28bd78741a139d53ee51
I20260812 02:34:51.610870 30351 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d92eab9524324b82b329fd272df49540: Bootstrap starting.
I20260812 02:34:51.613320 30351 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d92eab9524324b82b329fd272df49540: Neither blocks nor log segments found. Creating new log.
I20260812 02:34:51.614230 30351 log.cc:824] T 00000000000000000000000000000000 P d92eab9524324b82b329fd272df49540: Log is configured to *not* fsync() on all Append() calls
Aug 12 02:34:51 dist-test-slave-2w3w krb5kdc[30263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502091, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.103.62@KRBTEST.COM
I20260812 02:34:51.616716 30351 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d92eab9524324b82b329fd272df49540: No bootstrap required, opened a new log
I20260812 02:34:51.621093 30351 raft_consensus.cc:348] T 00000000000000000000000000000000 P d92eab9524324b82b329fd272df49540 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d92eab9524324b82b329fd272df49540" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 43307 } }
I20260812 02:34:51.621399 30351 raft_consensus.cc:374] T 00000000000000000000000000000000 P d92eab9524324b82b329fd272df49540 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:34:51.621467 30351 raft_consensus.cc:729] T 00000000000000000000000000000000 P d92eab9524324b82b329fd272df49540 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d92eab9524324b82b329fd272df49540, State: Initialized, Role: FOLLOWER
I20260812 02:34:51.621991 30351 consensus_queue.cc:260] T 00000000000000000000000000000000 P d92eab9524324b82b329fd272df49540 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d92eab9524324b82b329fd272df49540" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 43307 } }
I20260812 02:34:51.622126 30351 raft_consensus.cc:388] T 00000000000000000000000000000000 P d92eab9524324b82b329fd272df49540 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 02:34:51.622211 30351 raft_consensus.cc:482] T 00000000000000000000000000000000 P d92eab9524324b82b329fd272df49540 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 02:34:51.622326 30351 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d92eab9524324b82b329fd272df49540 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:34:51.622480 30354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:51.608452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32990 (local address 127.14.103.62:43307)
0812 02:34:51.609025 (+ 573us) server_negotiation.cc:206] Beginning negotiation
0812 02:34:51.609046 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0812 02:34:51.609093 (+ 47us) server_negotiation.cc:407] Connection header received
0812 02:34:51.609911 (+ 818us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:34:51.609930 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:34:51.610313 (+ 383us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:34:51.610676 (+ 363us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:34:51.611568 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:51.612599 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:51.613485 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:51.614066 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:51.614245 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:51.616910 (+ 2665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:34:51.616915 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:34:51.616930 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:34:51.616959 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:34:51.618808 (+ 1849us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:34:51.619734 (+ 926us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:34:51.619741 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:34:51.619747 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:34:51.619892 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:34:51.620190 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:34:51.620194 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:34:51.620196 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:34:51.620713 (+ 517us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:34:51.620946 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0812 02:34:51.621200 (+ 254us) server_negotiation.cc:299] Negotiation successful
0812 02:34:51.621562 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":168,"threads_started":1}
I20260812 02:34:51.623468 30351 raft_consensus.cc:504] T 00000000000000000000000000000000 P d92eab9524324b82b329fd272df49540 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d92eab9524324b82b329fd272df49540" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 43307 } }
I20260812 02:34:51.623829 30351 leader_election.cc:302] T 00000000000000000000000000000000 P d92eab9524324b82b329fd272df49540 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d92eab9524324b82b329fd272df49540; no voters:
I20260812 02:34:51.624179 30351 leader_election.cc:288] T 00000000000000000000000000000000 P d92eab9524324b82b329fd272df49540 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 02:34:51.624274 30356 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d92eab9524324b82b329fd272df49540 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:34:51.624456 30356 raft_consensus.cc:686] T 00000000000000000000000000000000 P d92eab9524324b82b329fd272df49540 [term 1 LEADER]: Becoming Leader. State: Replica: d92eab9524324b82b329fd272df49540, State: Running, Role: LEADER
I20260812 02:34:51.624765 30356 consensus_queue.cc:237] T 00000000000000000000000000000000 P d92eab9524324b82b329fd272df49540 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d92eab9524324b82b329fd272df49540" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 43307 } }
I20260812 02:34:51.625548 30351 sys_catalog.cc:564] T 00000000000000000000000000000000 P d92eab9524324b82b329fd272df49540 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:34:51.625967 30356 sys_catalog.cc:455] T 00000000000000000000000000000000 P d92eab9524324b82b329fd272df49540 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d92eab9524324b82b329fd272df49540. Latest consensus state: current_term: 1 leader_uuid: "d92eab9524324b82b329fd272df49540" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d92eab9524324b82b329fd272df49540" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 43307 } } }
I20260812 02:34:51.626086 30356 sys_catalog.cc:458] T 00000000000000000000000000000000 P d92eab9524324b82b329fd272df49540 [sys.catalog]: This master's current role is: LEADER
I20260812 02:34:51.626579 30357 sys_catalog.cc:455] T 00000000000000000000000000000000 P d92eab9524324b82b329fd272df49540 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d92eab9524324b82b329fd272df49540" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d92eab9524324b82b329fd272df49540" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 43307 } } }
I20260812 02:34:51.626722 30357 sys_catalog.cc:458] T 00000000000000000000000000000000 P d92eab9524324b82b329fd272df49540 [sys.catalog]: This master's current role is: LEADER
I20260812 02:34:51.627297 30362 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 02:34:51.630184 30362 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 02:34:51.636168 30362 catalog_manager.cc:1347] Generated new cluster ID: fdb7ae612c83410bbce5c0d124a4977b
I20260812 02:34:51.636268 30362 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 02:34:51.646765 30362 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 02:34:51.647825 30362 catalog_manager.cc:1504] Loading token signing keys...
I20260812 02:34:51.661962 30362 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d92eab9524324b82b329fd272df49540: Generated new TSK 0
I20260812 02:34:51.662942 30362 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.103.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.103.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.103.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.1.keytab.
Entry for principal kudu/127.14.103.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.1.keytab.
I20260812 02:34:51.699703 14748 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbep4M4/build/debug/bin/kudu
/tmp/dist-test-taskbep4M4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.103.1:0
--local_ip_for_outbound_sockets=127.14.103.1
--webserver_interface=127.14.103.1
--webserver_port=0
--tserver_master_addrs=127.14.103.62:43307
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.1.keytab
--principal=kudu/127.14.103.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.103.20:36423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:34:51.822111 30385 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:34:51.822431 30385 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:34:51.822494 30385 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:34:51.827448 30385 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:34:51.827526 30385 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:34:51.827592 30385 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.103.1
I20260812 02:34:51.833803 30385 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.103.20:36423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.1.keytab
--principal=kudu/127.14.103.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.103.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.103.1
--webserver_port=0
--tserver_master_addrs=127.14.103.62:43307
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30385
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.103.1
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:34:51.835112 30385 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:34:51.836208 30385 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:34:51.847011 30395 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:34:51.847018 30394 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:34:51.847011 30397 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:34:51.847543 30385 server_base.cc:1034] running on GCE node
I20260812 02:34:51.848001 30385 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:34:51.848742 30385 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:34:51.849953 30385 hybrid_clock.cc:648] HybridClock initialized: now 1786502091849916 us; error 53 us; skew 500 ppm
Aug 12 02:34:51 dist-test-slave-2w3w krb5kdc[30263](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502091, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:34:51.854000 30385 init.cc:377] Logged in from keytab as kudu/127.14.103.1@KRBTEST.COM (short username kudu)
I20260812 02:34:51.855216 30385 webserver.cc:460] Webserver started at http://127.14.103.1:44913/ using document root <none> and password file <none>
I20260812 02:34:51.855815 30385 fs_manager.cc:359] Metadata directory not provided
I20260812 02:34:51.855896 30385 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:34:51.856081 30385 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:34:51.858251 30385 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/minicluster-data/ts-0/data/instance:
uuid: "649a01114e0a46628578bff8843a050a"
format_stamp: "Formatted at 2026-08-12 02:34:51 on dist-test-slave-2w3w"
server_key: "635812a0cc12b707e20ed3264aacc3ea"
server_key_iv: "37d30bf5c49236f63ef0991a57da38d0"
server_key_version: "encryptionkey@0"
I20260812 02:34:51.858855 30385 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/minicluster-data/ts-0/wal/instance:
uuid: "649a01114e0a46628578bff8843a050a"
format_stamp: "Formatted at 2026-08-12 02:34:51 on dist-test-slave-2w3w"
server_key: "635812a0cc12b707e20ed3264aacc3ea"
server_key_iv: "37d30bf5c49236f63ef0991a57da38d0"
server_key_version: "encryptionkey@0"
I20260812 02:34:51.863112 30385 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260812 02:34:51.865929 30405 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:34:51.867666 30385 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260812 02:34:51.867810 30385 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/minicluster-data/ts-0/wal
uuid: "649a01114e0a46628578bff8843a050a"
format_stamp: "Formatted at 2026-08-12 02:34:51 on dist-test-slave-2w3w"
server_key: "635812a0cc12b707e20ed3264aacc3ea"
server_key_iv: "37d30bf5c49236f63ef0991a57da38d0"
server_key_version: "encryptionkey@0"
I20260812 02:34:51.867947 30385 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:34:51.888060 30385 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:34:51.891644 30385 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:34:51.891906 30385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:34:51.892590 30385 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:34:51.893527 30385 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:34:51.893579 30385 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:34:51.893651 30385 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:34:51.893688 30385 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:34:51.903863 30385 rpc_server.cc:307] RPC server started. Bound to: 127.14.103.1:45955
I20260812 02:34:51.903882 30538 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.103.1:45955 every 8 connection(s)
I20260812 02:34:51.904870 30385 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/minicluster-data/ts-0/data/info.pb
I20260812 02:34:51.908458 14748 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbep4M4/build/debug/bin/kudu as pid 30385
I20260812 02:34:51.908594 14748 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/minicluster-data/ts-0/wal/instance
I20260812 02:34:51.908865 14748 external_mini_cluster.cc:1428] Setting key 4972388ae6389d2dc824f90c6086e9c0
Aug 12 02:34:51 dist-test-slave-2w3w krb5kdc[30263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502091, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.1@KRBTEST.COM for kudu/127.14.103.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.14.103.2@KRBTEST.COM; defaulting to no policy
I20260812 02:34:51.920987 30354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:51.906785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.1:47043 (local address 127.14.103.62:43307)
0812 02:34:51.907028 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0812 02:34:51.907032 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:34:51.907853 (+ 821us) server_negotiation.cc:407] Connection header received
0812 02:34:51.909461 (+ 1608us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:34:51.909464 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:34:51.909516 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:34:51.909595 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:34:51.911024 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:51.911876 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:51.913105 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:51.913514 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:51.913638 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:51.916721 (+ 3083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:34:51.916724 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:34:51.916726 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:34:51.916754 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:34:51.918463 (+ 1709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:34:51.919161 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:34:51.919165 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:34:51.919167 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:34:51.919260 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:34:51.919674 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:34:51.919678 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:34:51.919680 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:34:51.919900 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:34:51.920020 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0812 02:34:51.920636 (+ 616us) server_negotiation.cc:299] Negotiation successful
0812 02:34:51.920797 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260812 02:34:51.922042 30543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:51.907221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:43307 (local address 127.14.103.1:47043)
0812 02:34:51.907695 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0812 02:34:51.907739 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0812 02:34:51.909197 (+ 1458us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:34:51.909809 (+ 612us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:34:51.909817 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:34:51.910275 (+ 458us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:34:51.910816 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:34:51.910830 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:34:51.912026 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:34:51.912028 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:34:51.912973 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:34:51.912984 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:34:51.913685 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:34:51.913687 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:34:51.913841 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:51.914591 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:34:51.914619 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:34:51.916519 (+ 1900us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:34:51.918630 (+ 2111us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:34:51.918638 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:34:51.918659 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:34:51.919043 (+ 384us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:34:51.919376 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:34:51.919380 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:34:51.919382 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:34:51.919557 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:34:51.920021 (+ 464us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:34:51.920030 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:34:51.920332 (+ 302us) client_negotiation.cc:769] Sending connection context
0812 02:34:51.920573 (+ 241us) client_negotiation.cc:240] Negotiation successful
0812 02:34:51.921037 (+ 464us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":125,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.103.2@KRBTEST.COM" created.
I20260812 02:34:51.923638 30539 heartbeater.cc:347] Connected to a master server at 127.14.103.62:43307
I20260812 02:34:51.923925 30539 heartbeater.cc:464] Registering TS with master...
I20260812 02:34:51.924470 30539 heartbeater.cc:511] Master 127.14.103.62:43307 requested a full tablet report, sending...
I20260812 02:34:51.926172 30313 ts_manager.cc:194] Registered new tserver with Master: 649a01114e0a46628578bff8843a050a (127.14.103.1:45955)
I20260812 02:34:51.927654 30313 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.103.1@KRBTEST.COM'} at 127.14.103.1:47043
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.103.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.2.keytab.
Entry for principal kudu/127.14.103.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.2.keytab.
I20260812 02:34:51.943085 14748 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbep4M4/build/debug/bin/kudu
/tmp/dist-test-taskbep4M4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.103.2:0
--local_ip_for_outbound_sockets=127.14.103.2
--webserver_interface=127.14.103.2
--webserver_port=0
--tserver_master_addrs=127.14.103.62:43307
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.2.keytab
--principal=kudu/127.14.103.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.103.20:36423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:34:52.065265 30547 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:34:52.065610 30547 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:34:52.065689 30547 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:34:52.070189 30547 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:34:52.070291 30547 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:34:52.070381 30547 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.103.2
I20260812 02:34:52.076375 30547 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.103.20:36423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.2.keytab
--principal=kudu/127.14.103.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.103.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.103.2
--webserver_port=0
--tserver_master_addrs=127.14.103.62:43307
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30547
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.103.2
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:34:52.077610 30547 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:34:52.078685 30547 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:34:52.089437 30553 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:34:52.089429 30552 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:34:52.089442 30555 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:34:52.089650 30547 server_base.cc:1034] running on GCE node
I20260812 02:34:52.090266 30547 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:34:52.091112 30547 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:34:52.092326 30547 hybrid_clock.cc:648] HybridClock initialized: now 1786502092092289 us; error 54 us; skew 500 ppm
Aug 12 02:34:52 dist-test-slave-2w3w krb5kdc[30263](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502092, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:34:52.096081 30547 init.cc:377] Logged in from keytab as kudu/127.14.103.2@KRBTEST.COM (short username kudu)
I20260812 02:34:52.097126 30547 webserver.cc:460] Webserver started at http://127.14.103.2:34039/ using document root <none> and password file <none>
I20260812 02:34:52.097776 30547 fs_manager.cc:359] Metadata directory not provided
I20260812 02:34:52.097882 30547 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:34:52.098119 30547 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:34:52.100397 30547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/minicluster-data/ts-1/data/instance:
uuid: "3909f7c94d654754a9d589bb0ae09e8a"
format_stamp: "Formatted at 2026-08-12 02:34:52 on dist-test-slave-2w3w"
server_key: "2f5c8300d53314fffe9ae0ed32f4ad3e"
server_key_iv: "d2bd2e696ab65ca38afef510784b98ef"
server_key_version: "encryptionkey@0"
I20260812 02:34:52.101087 30547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/minicluster-data/ts-1/wal/instance:
uuid: "3909f7c94d654754a9d589bb0ae09e8a"
format_stamp: "Formatted at 2026-08-12 02:34:52 on dist-test-slave-2w3w"
server_key: "2f5c8300d53314fffe9ae0ed32f4ad3e"
server_key_iv: "d2bd2e696ab65ca38afef510784b98ef"
server_key_version: "encryptionkey@0"
I20260812 02:34:52.105500 30547 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260812 02:34:52.108493 30562 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:34:52.109964 30547 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260812 02:34:52.110133 30547 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/minicluster-data/ts-1/wal
uuid: "3909f7c94d654754a9d589bb0ae09e8a"
format_stamp: "Formatted at 2026-08-12 02:34:52 on dist-test-slave-2w3w"
server_key: "2f5c8300d53314fffe9ae0ed32f4ad3e"
server_key_iv: "d2bd2e696ab65ca38afef510784b98ef"
server_key_version: "encryptionkey@0"
I20260812 02:34:52.110268 30547 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:34:52.124924 30547 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:34:52.128870 30547 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:34:52.129103 30547 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:34:52.129760 30547 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:34:52.130710 30547 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:34:52.130796 30547 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:34:52.130892 30547 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:34:52.130931 30547 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:34:52.140380 30547 rpc_server.cc:307] RPC server started. Bound to: 127.14.103.2:46173
I20260812 02:34:52.140422 30688 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.103.2:46173 every 8 connection(s)
I20260812 02:34:52.141389 30547 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/minicluster-data/ts-1/data/info.pb
I20260812 02:34:52.141848 14748 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbep4M4/build/debug/bin/kudu as pid 30547
I20260812 02:34:52.141970 14748 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/minicluster-data/ts-1/wal/instance
I20260812 02:34:52.142294 14748 external_mini_cluster.cc:1428] Setting key 0576a92aff193ed5d4b0cac718de8714
Aug 12 02:34:52 dist-test-slave-2w3w krb5kdc[30263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502092, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.2@KRBTEST.COM for kudu/127.14.103.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.14.103.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.103.3@KRBTEST.COM" created.
I20260812 02:34:52.159142 30354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:52.143661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.2:39635 (local address 127.14.103.62:43307)
0812 02:34:52.143796 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0812 02:34:52.143801 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:34:52.144976 (+ 1175us) server_negotiation.cc:407] Connection header received
0812 02:34:52.146579 (+ 1603us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:34:52.146582 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:34:52.146632 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:34:52.146704 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:34:52.148568 (+ 1864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:52.149107 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:52.151121 (+ 2014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:52.151380 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:52.151475 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:52.154563 (+ 3088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:34:52.154566 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:34:52.154568 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:34:52.154596 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:34:52.156528 (+ 1932us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:34:52.157215 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:34:52.157221 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:34:52.157223 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:34:52.157317 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:34:52.157722 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:34:52.157729 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:34:52.157733 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:34:52.157966 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:34:52.158089 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0812 02:34:52.158656 (+ 567us) server_negotiation.cc:299] Negotiation successful
0812 02:34:52.158836 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260812 02:34:52.159986 30692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:52.144147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:43307 (local address 127.14.103.2:39635)
0812 02:34:52.144732 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0812 02:34:52.144845 (+ 113us) client_negotiation.cc:174] Beginning negotiation
0812 02:34:52.146316 (+ 1471us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:34:52.146924 (+ 608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:34:52.146935 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:34:52.147572 (+ 637us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:34:52.148371 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:34:52.148395 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:34:52.149277 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:34:52.149284 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:34:52.150930 (+ 1646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:34:52.150947 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:34:52.151571 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:34:52.151578 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:34:52.151831 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:52.152544 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:34:52.152571 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:34:52.154373 (+ 1802us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:34:52.156687 (+ 2314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:34:52.156694 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:34:52.156715 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:34:52.157093 (+ 378us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:34:52.157460 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:34:52.157463 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:34:52.157465 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:34:52.157617 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:34:52.158101 (+ 484us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:34:52.158107 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:34:52.158389 (+ 282us) client_negotiation.cc:769] Sending connection context
0812 02:34:52.158571 (+ 182us) client_negotiation.cc:240] Negotiation successful
0812 02:34:52.158887 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":167,"threads_started":1}
I20260812 02:34:52.161396 30690 heartbeater.cc:347] Connected to a master server at 127.14.103.62:43307
I20260812 02:34:52.161664 30690 heartbeater.cc:464] Registering TS with master...
I20260812 02:34:52.162237 30690 heartbeater.cc:511] Master 127.14.103.62:43307 requested a full tablet report, sending...
I20260812 02:34:52.163414 30313 ts_manager.cc:194] Registered new tserver with Master: 3909f7c94d654754a9d589bb0ae09e8a (127.14.103.2:46173)
I20260812 02:34:52.164093 30313 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.103.2@KRBTEST.COM'} at 127.14.103.2:39635
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.103.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.3.keytab.
Entry for principal kudu/127.14.103.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.3.keytab.
I20260812 02:34:52.178598 14748 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbep4M4/build/debug/bin/kudu
/tmp/dist-test-taskbep4M4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.103.3:0
--local_ip_for_outbound_sockets=127.14.103.3
--webserver_interface=127.14.103.3
--webserver_port=0
--tserver_master_addrs=127.14.103.62:43307
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.3.keytab
--principal=kudu/127.14.103.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.103.20:36423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:34:52.300263 30696 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:34:52.300616 30696 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:34:52.300689 30696 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:34:52.305100 30696 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:34:52.305213 30696 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:34:52.305313 30696 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.103.3
I20260812 02:34:52.311160 30696 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.103.20:36423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.3.keytab
--principal=kudu/127.14.103.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.103.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.103.3
--webserver_port=0
--tserver_master_addrs=127.14.103.62:43307
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30696
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.103.3
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:34:52.312356 30696 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:34:52.313406 30696 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:34:52.324204 30702 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:34:52.324200 30703 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:34:52.324194 30707 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:34:52.324523 30696 server_base.cc:1034] running on GCE node
I20260812 02:34:52.325129 30696 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:34:52.325877 30696 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:34:52.327057 30696 hybrid_clock.cc:648] HybridClock initialized: now 1786502092327028 us; error 47 us; skew 500 ppm
Aug 12 02:34:52 dist-test-slave-2w3w krb5kdc[30263](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502092, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:34:52.330446 30696 init.cc:377] Logged in from keytab as kudu/127.14.103.3@KRBTEST.COM (short username kudu)
I20260812 02:34:52.331632 30696 webserver.cc:460] Webserver started at http://127.14.103.3:33687/ using document root <none> and password file <none>
I20260812 02:34:52.332326 30696 fs_manager.cc:359] Metadata directory not provided
I20260812 02:34:52.332435 30696 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:34:52.332706 30696 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:34:52.335270 30696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/minicluster-data/ts-2/data/instance:
uuid: "bfbbe1513aad4179abc52064d9250a9a"
format_stamp: "Formatted at 2026-08-12 02:34:52 on dist-test-slave-2w3w"
server_key: "5a89d8ac702231998b7f2134835d7d8c"
server_key_iv: "ab713679a6525a8ff3e27f67fb2b5a2b"
server_key_version: "encryptionkey@0"
I20260812 02:34:52.336042 30696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/minicluster-data/ts-2/wal/instance:
uuid: "bfbbe1513aad4179abc52064d9250a9a"
format_stamp: "Formatted at 2026-08-12 02:34:52 on dist-test-slave-2w3w"
server_key: "5a89d8ac702231998b7f2134835d7d8c"
server_key_iv: "ab713679a6525a8ff3e27f67fb2b5a2b"
server_key_version: "encryptionkey@0"
I20260812 02:34:52.340728 30696 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260812 02:34:52.344043 30715 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:34:52.345600 30696 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 02:34:52.345786 30696 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/minicluster-data/ts-2/wal
uuid: "bfbbe1513aad4179abc52064d9250a9a"
format_stamp: "Formatted at 2026-08-12 02:34:52 on dist-test-slave-2w3w"
server_key: "5a89d8ac702231998b7f2134835d7d8c"
server_key_iv: "ab713679a6525a8ff3e27f67fb2b5a2b"
server_key_version: "encryptionkey@0"
I20260812 02:34:52.345942 30696 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:34:52.356379 30696 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:34:52.359606 30696 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:34:52.359843 30696 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:34:52.360469 30696 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:34:52.361338 30696 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:34:52.361410 30696 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:34:52.361478 30696 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:34:52.361523 30696 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:34:52.371577 30696 rpc_server.cc:307] RPC server started. Bound to: 127.14.103.3:32843
I20260812 02:34:52.371593 30842 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.103.3:32843 every 8 connection(s)
I20260812 02:34:52.372848 30696 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/minicluster-data/ts-2/data/info.pb
I20260812 02:34:52.377221 14748 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbep4M4/build/debug/bin/kudu as pid 30696
I20260812 02:34:52.377357 14748 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/minicluster-data/ts-2/wal/instance
I20260812 02:34:52.377640 14748 external_mini_cluster.cc:1428] Setting key 70a3f2865a081bb3a1550b1ea97757a6
Aug 12 02:34:52 dist-test-slave-2w3w krb5kdc[30263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502092, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.3@KRBTEST.COM for kudu/127.14.103.62@KRBTEST.COM
I20260812 02:34:52.389072 30354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:52.375020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.3:47235 (local address 127.14.103.62:43307)
0812 02:34:52.375184 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0812 02:34:52.375188 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:34:52.376108 (+ 920us) server_negotiation.cc:407] Connection header received
0812 02:34:52.377935 (+ 1827us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:34:52.377939 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:34:52.377993 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:34:52.378106 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:34:52.379875 (+ 1769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:52.380481 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:52.381701 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:52.381937 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:52.382025 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:52.384719 (+ 2694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:34:52.384722 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:34:52.384724 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:34:52.384752 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:34:52.386525 (+ 1773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:34:52.387305 (+ 780us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:34:52.387308 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:34:52.387310 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:34:52.387405 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:34:52.387747 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:34:52.387750 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:34:52.387751 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:34:52.388016 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:34:52.388123 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0812 02:34:52.388779 (+ 656us) server_negotiation.cc:299] Negotiation successful
0812 02:34:52.388933 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260812 02:34:52.389966 30845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:52.375381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:43307 (local address 127.14.103.3:47235)
0812 02:34:52.375923 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0812 02:34:52.375971 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0812 02:34:52.377654 (+ 1683us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:34:52.378424 (+ 770us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:34:52.378433 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:34:52.379076 (+ 643us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:34:52.379695 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:34:52.379709 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:34:52.380610 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:34:52.380613 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:34:52.381575 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:34:52.381585 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:34:52.382112 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:34:52.382115 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:34:52.382285 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:52.383024 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:34:52.383048 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:34:52.384539 (+ 1491us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:34:52.386681 (+ 2142us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:34:52.386687 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:34:52.386702 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:34:52.387180 (+ 478us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:34:52.387493 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:34:52.387495 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:34:52.387496 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:34:52.387656 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:34:52.388149 (+ 493us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:34:52.388171 (+ 22us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:34:52.388473 (+ 302us) client_negotiation.cc:769] Sending connection context
0812 02:34:52.388661 (+ 188us) client_negotiation.cc:240] Negotiation successful
0812 02:34:52.389001 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":118,"threads_started":1}
I20260812 02:34:52.391455 30843 heartbeater.cc:347] Connected to a master server at 127.14.103.62:43307
I20260812 02:34:52.391727 30843 heartbeater.cc:464] Registering TS with master...
I20260812 02:34:52.392256 30843 heartbeater.cc:511] Master 127.14.103.62:43307 requested a full tablet report, sending...
I20260812 02:34:52.393414 30312 ts_manager.cc:194] Registered new tserver with Master: bfbbe1513aad4179abc52064d9250a9a (127.14.103.3:32843)
I20260812 02:34:52.394027 30312 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.103.3@KRBTEST.COM'} at 127.14.103.3:47235
I20260812 02:34:52.404246 14748 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 02:34:52.415052 30354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:52.406638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32992 (local address 127.14.103.62:43307)
0812 02:34:52.406816 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0812 02:34:52.406830 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0812 02:34:52.406946 (+ 116us) server_negotiation.cc:407] Connection header received
0812 02:34:52.407104 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:34:52.407107 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:34:52.407158 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:34:52.407237 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:34:52.408057 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:52.408608 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:52.409445 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:52.409662 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:52.409752 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:52.411319 (+ 1567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:34:52.411322 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:34:52.411324 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:34:52.411355 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:34:52.413160 (+ 1805us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:34:52.413690 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:34:52.413693 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:34:52.413694 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:34:52.413784 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:34:52.414039 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:34:52.414041 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:34:52.414042 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:34:52.414358 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:34:52.414479 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0812 02:34:52.414700 (+ 221us) server_negotiation.cc:299] Negotiation successful
0812 02:34:52.414823 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260812 02:34:52.418604 30312 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32992:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 02:34:52.421789 30312 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 02:34:52.438197 30857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:52.431416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.3:32843 (local address 127.0.0.1:58266)
0812 02:34:52.432239 (+ 823us) negotiation.cc:107] Waiting for socket to connect
0812 02:34:52.432299 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0812 02:34:52.432446 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:34:52.433100 (+ 654us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:34:52.433109 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:34:52.433137 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:34:52.433315 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:34:52.433322 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:34:52.434689 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:34:52.434694 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:34:52.436184 (+ 1490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:34:52.436195 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:34:52.437477 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:34:52.437481 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:34:52.437679 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:52.437738 (+ 59us) client_negotiation.cc:769] Sending connection context
0812 02:34:52.437883 (+ 145us) client_negotiation.cc:240] Negotiation successful
0812 02:34:52.438014 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":667,"mutex_wait_us":169,"spinlock_wait_cycles":5248,"thread_start_us":108,"threads_started":1}
I20260812 02:34:52.438580 30861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:52.432198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.2:46173 (local address 127.0.0.1:39610)
0812 02:34:52.433076 (+ 878us) negotiation.cc:107] Waiting for socket to connect
0812 02:34:52.433091 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 02:34:52.433162 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:34:52.433884 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:34:52.433887 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:34:52.433906 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:34:52.434002 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:34:52.434008 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:34:52.435370 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:34:52.435373 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:34:52.436838 (+ 1465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:34:52.436848 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:34:52.437920 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:34:52.437924 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:34:52.438027 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:52.438050 (+ 23us) client_negotiation.cc:769] Sending connection context
0812 02:34:52.438320 (+ 270us) client_negotiation.cc:240] Negotiation successful
0812 02:34:52.438437 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":800,"mutex_wait_us":248,"thread_start_us":147,"threads_started":1}
I20260812 02:34:52.438701 30860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:52.431682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54378 (local address 127.14.103.1:45955)
0812 02:34:52.432744 (+ 1062us) server_negotiation.cc:206] Beginning negotiation
0812 02:34:52.432751 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 02:34:52.432827 (+ 76us) server_negotiation.cc:407] Connection header received
0812 02:34:52.432950 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:34:52.432988 (+ 38us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:34:52.433190 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:34:52.433641 (+ 451us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:34:52.433795 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:52.435015 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:52.436541 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:52.437461 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:52.437982 (+ 521us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:52.438126 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0812 02:34:52.438182 (+ 56us) server_negotiation.cc:299] Negotiation successful
0812 02:34:52.438558 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":949,"thread_start_us":135,"threads_started":1}
I20260812 02:34:52.438591 30858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:52.431416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.1:45955 (local address 127.0.0.1:54378)
0812 02:34:52.432239 (+ 823us) negotiation.cc:107] Waiting for socket to connect
0812 02:34:52.432300 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0812 02:34:52.432446 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:34:52.433343 (+ 897us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:34:52.433345 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:34:52.433364 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:34:52.433490 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:34:52.433498 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:34:52.435149 (+ 1651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:34:52.435153 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:34:52.436395 (+ 1242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:34:52.436404 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:34:52.437629 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:34:52.437633 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:34:52.437750 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:52.437768 (+ 18us) client_negotiation.cc:769] Sending connection context
0812 02:34:52.437841 (+ 73us) client_negotiation.cc:240] Negotiation successful
0812 02:34:52.438446 (+ 605us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":637,"mutex_wait_us":502,"spinlock_wait_cycles":3968,"thread_start_us":77,"threads_started":1}
I20260812 02:34:52.439102 30859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:52.431581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58266 (local address 127.14.103.3:32843)
0812 02:34:52.432394 (+ 813us) server_negotiation.cc:206] Beginning negotiation
0812 02:34:52.432408 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0812 02:34:52.432435 (+ 27us) server_negotiation.cc:407] Connection header received
0812 02:34:52.432628 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:34:52.432635 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:34:52.432861 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:34:52.433533 (+ 672us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:34:52.433659 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:52.434560 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:52.436353 (+ 1793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:52.437311 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:52.437434 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:52.437603 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0812 02:34:52.438660 (+ 1057us) server_negotiation.cc:299] Negotiation successful
0812 02:34:52.438847 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":695,"thread_start_us":199,"threads_started":1}
I20260812 02:34:52.439412 30862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:52.432823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39610 (local address 127.14.103.2:46173)
0812 02:34:52.433441 (+ 618us) server_negotiation.cc:206] Beginning negotiation
0812 02:34:52.433454 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0812 02:34:52.433477 (+ 23us) server_negotiation.cc:407] Connection header received
0812 02:34:52.433553 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:34:52.433560 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:34:52.433728 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:34:52.433885 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:34:52.434123 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:52.435227 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:52.436966 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:52.437788 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:52.437915 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:52.438063 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0812 02:34:52.438928 (+ 865us) server_negotiation.cc:299] Negotiation successful
0812 02:34:52.439244 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":522,"thread_start_us":146,"threads_started":1}
I20260812 02:34:52.442081 30769 tablet_service.cc:1467] Processing CreateTablet for tablet 01c1cc2389af4f9198c7ec1c7a871ff0 (DEFAULT_TABLE table=test-table [id=444bccb026034de8b4a56c055422dade]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:34:52.442279 30616 tablet_service.cc:1467] Processing CreateTablet for tablet 01c1cc2389af4f9198c7ec1c7a871ff0 (DEFAULT_TABLE table=test-table [id=444bccb026034de8b4a56c055422dade]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:34:52.442489 30460 tablet_service.cc:1467] Processing CreateTablet for tablet 01c1cc2389af4f9198c7ec1c7a871ff0 (DEFAULT_TABLE table=test-table [id=444bccb026034de8b4a56c055422dade]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:34:52.443393 30769 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01c1cc2389af4f9198c7ec1c7a871ff0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:34:52.443631 30460 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01c1cc2389af4f9198c7ec1c7a871ff0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:34:52.443632 30616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01c1cc2389af4f9198c7ec1c7a871ff0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:34:52.451503 30863 tablet_bootstrap.cc:492] T 01c1cc2389af4f9198c7ec1c7a871ff0 P bfbbe1513aad4179abc52064d9250a9a: Bootstrap starting.
I20260812 02:34:52.453280 30866 tablet_bootstrap.cc:492] T 01c1cc2389af4f9198c7ec1c7a871ff0 P 649a01114e0a46628578bff8843a050a: Bootstrap starting.
I20260812 02:34:52.454442 30864 tablet_bootstrap.cc:492] T 01c1cc2389af4f9198c7ec1c7a871ff0 P 3909f7c94d654754a9d589bb0ae09e8a: Bootstrap starting.
I20260812 02:34:52.457114 30866 tablet_bootstrap.cc:654] T 01c1cc2389af4f9198c7ec1c7a871ff0 P 649a01114e0a46628578bff8843a050a: Neither blocks nor log segments found. Creating new log.
I20260812 02:34:52.457621 30864 tablet_bootstrap.cc:654] T 01c1cc2389af4f9198c7ec1c7a871ff0 P 3909f7c94d654754a9d589bb0ae09e8a: Neither blocks nor log segments found. Creating new log.
I20260812 02:34:52.457641 30863 tablet_bootstrap.cc:654] T 01c1cc2389af4f9198c7ec1c7a871ff0 P bfbbe1513aad4179abc52064d9250a9a: Neither blocks nor log segments found. Creating new log.
I20260812 02:34:52.458230 30866 log.cc:824] T 01c1cc2389af4f9198c7ec1c7a871ff0 P 649a01114e0a46628578bff8843a050a: Log is configured to *not* fsync() on all Append() calls
I20260812 02:34:52.458433 30864 log.cc:824] T 01c1cc2389af4f9198c7ec1c7a871ff0 P 3909f7c94d654754a9d589bb0ae09e8a: Log is configured to *not* fsync() on all Append() calls
I20260812 02:34:52.459601 30863 log.cc:824] T 01c1cc2389af4f9198c7ec1c7a871ff0 P bfbbe1513aad4179abc52064d9250a9a: Log is configured to *not* fsync() on all Append() calls
I20260812 02:34:52.460862 30864 tablet_bootstrap.cc:492] T 01c1cc2389af4f9198c7ec1c7a871ff0 P 3909f7c94d654754a9d589bb0ae09e8a: No bootstrap required, opened a new log
I20260812 02:34:52.461125 30864 ts_tablet_manager.cc:1397] T 01c1cc2389af4f9198c7ec1c7a871ff0 P 3909f7c94d654754a9d589bb0ae09e8a: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260812 02:34:52.461892 30866 tablet_bootstrap.cc:492] T 01c1cc2389af4f9198c7ec1c7a871ff0 P 649a01114e0a46628578bff8843a050a: No bootstrap required, opened a new log
I20260812 02:34:52.462141 30866 ts_tablet_manager.cc:1397] T 01c1cc2389af4f9198c7ec1c7a871ff0 P 649a01114e0a46628578bff8843a050a: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260812 02:34:52.462749 30863 tablet_bootstrap.cc:492] T 01c1cc2389af4f9198c7ec1c7a871ff0 P bfbbe1513aad4179abc52064d9250a9a: No bootstrap required, opened a new log
I20260812 02:34:52.463004 30863 ts_tablet_manager.cc:1397] T 01c1cc2389af4f9198c7ec1c7a871ff0 P bfbbe1513aad4179abc52064d9250a9a: Time spent bootstrapping tablet: real 0.012s user 0.010s sys 0.000s
I20260812 02:34:52.465584 30864 raft_consensus.cc:348] T 01c1cc2389af4f9198c7ec1c7a871ff0 P 3909f7c94d654754a9d589bb0ae09e8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfbbe1513aad4179abc52064d9250a9a" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 32843 } } peers { permanent_uuid: "649a01114e0a46628578bff8843a050a" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 45955 } } peers { permanent_uuid: "3909f7c94d654754a9d589bb0ae09e8a" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 46173 } }
I20260812 02:34:52.465700 30866 raft_consensus.cc:348] T 01c1cc2389af4f9198c7ec1c7a871ff0 P 649a01114e0a46628578bff8843a050a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfbbe1513aad4179abc52064d9250a9a" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 32843 } } peers { permanent_uuid: "649a01114e0a46628578bff8843a050a" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 45955 } } peers { permanent_uuid: "3909f7c94d654754a9d589bb0ae09e8a" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 46173 } }
I20260812 02:34:52.465930 30864 raft_consensus.cc:374] T 01c1cc2389af4f9198c7ec1c7a871ff0 P 3909f7c94d654754a9d589bb0ae09e8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:34:52.465958 30866 raft_consensus.cc:374] T 01c1cc2389af4f9198c7ec1c7a871ff0 P 649a01114e0a46628578bff8843a050a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:34:52.466012 30866 raft_consensus.cc:729] T 01c1cc2389af4f9198c7ec1c7a871ff0 P 649a01114e0a46628578bff8843a050a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 649a01114e0a46628578bff8843a050a, State: Initialized, Role: FOLLOWER
I20260812 02:34:52.465996 30864 raft_consensus.cc:729] T 01c1cc2389af4f9198c7ec1c7a871ff0 P 3909f7c94d654754a9d589bb0ae09e8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3909f7c94d654754a9d589bb0ae09e8a, State: Initialized, Role: FOLLOWER
I20260812 02:34:52.466512 30866 consensus_queue.cc:260] T 01c1cc2389af4f9198c7ec1c7a871ff0 P 649a01114e0a46628578bff8843a050a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfbbe1513aad4179abc52064d9250a9a" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 32843 } } peers { permanent_uuid: "649a01114e0a46628578bff8843a050a" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 45955 } } peers { permanent_uuid: "3909f7c94d654754a9d589bb0ae09e8a" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 46173 } }
I20260812 02:34:52.466569 30864 consensus_queue.cc:260] T 01c1cc2389af4f9198c7ec1c7a871ff0 P 3909f7c94d654754a9d589bb0ae09e8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfbbe1513aad4179abc52064d9250a9a" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 32843 } } peers { permanent_uuid: "649a01114e0a46628578bff8843a050a" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 45955 } } peers { permanent_uuid: "3909f7c94d654754a9d589bb0ae09e8a" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 46173 } }
I20260812 02:34:52.466909 30863 raft_consensus.cc:348] T 01c1cc2389af4f9198c7ec1c7a871ff0 P bfbbe1513aad4179abc52064d9250a9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfbbe1513aad4179abc52064d9250a9a" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 32843 } } peers { permanent_uuid: "649a01114e0a46628578bff8843a050a" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 45955 } } peers { permanent_uuid: "3909f7c94d654754a9d589bb0ae09e8a" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 46173 } }
I20260812 02:34:52.467229 30863 raft_consensus.cc:374] T 01c1cc2389af4f9198c7ec1c7a871ff0 P bfbbe1513aad4179abc52064d9250a9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:34:52.467284 30863 raft_consensus.cc:729] T 01c1cc2389af4f9198c7ec1c7a871ff0 P bfbbe1513aad4179abc52064d9250a9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bfbbe1513aad4179abc52064d9250a9a, State: Initialized, Role: FOLLOWER
I20260812 02:34:52.467397 30539 heartbeater.cc:503] Master 127.14.103.62:43307 was elected leader, sending a full tablet report...
I20260812 02:34:52.467695 30866 ts_tablet_manager.cc:1428] T 01c1cc2389af4f9198c7ec1c7a871ff0 P 649a01114e0a46628578bff8843a050a: Time spent starting tablet: real 0.005s user 0.007s sys 0.000s
I20260812 02:34:52.467697 30864 ts_tablet_manager.cc:1428] T 01c1cc2389af4f9198c7ec1c7a871ff0 P 3909f7c94d654754a9d589bb0ae09e8a: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
I20260812 02:34:52.467732 30863 consensus_queue.cc:260] T 01c1cc2389af4f9198c7ec1c7a871ff0 P bfbbe1513aad4179abc52064d9250a9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfbbe1513aad4179abc52064d9250a9a" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 32843 } } peers { permanent_uuid: "649a01114e0a46628578bff8843a050a" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 45955 } } peers { permanent_uuid: "3909f7c94d654754a9d589bb0ae09e8a" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 46173 } }
I20260812 02:34:52.468050 30690 heartbeater.cc:503] Master 127.14.103.62:43307 was elected leader, sending a full tablet report...
I20260812 02:34:52.468465 30843 heartbeater.cc:503] Master 127.14.103.62:43307 was elected leader, sending a full tablet report...
I20260812 02:34:52.470889 30863 ts_tablet_manager.cc:1428] T 01c1cc2389af4f9198c7ec1c7a871ff0 P bfbbe1513aad4179abc52064d9250a9a: Time spent starting tablet: real 0.008s user 0.005s sys 0.000s
I20260812 02:34:52.510200 30871 raft_consensus.cc:482] T 01c1cc2389af4f9198c7ec1c7a871ff0 P 649a01114e0a46628578bff8843a050a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 02:34:52.510486 30871 raft_consensus.cc:504] T 01c1cc2389af4f9198c7ec1c7a871ff0 P 649a01114e0a46628578bff8843a050a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfbbe1513aad4179abc52064d9250a9a" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 32843 } } peers { permanent_uuid: "649a01114e0a46628578bff8843a050a" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 45955 } } peers { permanent_uuid: "3909f7c94d654754a9d589bb0ae09e8a" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 46173 } }
I20260812 02:34:52.511766 30871 leader_election.cc:288] T 01c1cc2389af4f9198c7ec1c7a871ff0 P 649a01114e0a46628578bff8843a050a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bfbbe1513aad4179abc52064d9250a9a (127.14.103.3:32843), 3909f7c94d654754a9d589bb0ae09e8a (127.14.103.2:46173)
I20260812 02:34:52.518378 30862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:52.512300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.1:54523 (local address 127.14.103.2:46173)
0812 02:34:52.512569 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0812 02:34:52.512573 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:34:52.512861 (+ 288us) server_negotiation.cc:407] Connection header received
0812 02:34:52.512976 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:34:52.512979 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:34:52.513038 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:34:52.513140 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:34:52.513853 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:52.514664 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:52.516885 (+ 2221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:52.517582 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:52.517728 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:52.517786 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0812 02:34:52.518168 (+ 382us) server_negotiation.cc:299] Negotiation successful
0812 02:34:52.518252 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178}
I20260812 02:34:52.518395 30859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:52.512309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.1:44025 (local address 127.14.103.3:32843)
0812 02:34:52.512508 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0812 02:34:52.512513 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:34:52.512815 (+ 302us) server_negotiation.cc:407] Connection header received
0812 02:34:52.512976 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:34:52.512979 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:34:52.513034 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:34:52.513137 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:34:52.513806 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:52.514703 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:52.516891 (+ 2188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:52.517582 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:52.517728 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:52.517801 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0812 02:34:52.518167 (+ 366us) server_negotiation.cc:299] Negotiation successful
0812 02:34:52.518265 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260812 02:34:52.518803 30874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:52.512154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.2:46173 (local address 127.14.103.1:54523)
0812 02:34:52.512742 (+ 588us) negotiation.cc:107] Waiting for socket to connect
0812 02:34:52.512756 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 02:34:52.512857 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:34:52.513229 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:34:52.513313 (+ 84us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:34:52.513358 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:34:52.513644 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:34:52.513653 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:34:52.514918 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:34:52.514922 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:34:52.516721 (+ 1799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:34:52.516735 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:34:52.517745 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:34:52.517750 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:34:52.517967 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:52.518023 (+ 56us) client_negotiation.cc:769] Sending connection context
0812 02:34:52.518580 (+ 557us) client_negotiation.cc:240] Negotiation successful
0812 02:34:52.518671 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":512,"mutex_wait_us":341,"spinlock_wait_cycles":22144,"thread_start_us":124,"threads_started":1}
I20260812 02:34:52.518430 30877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:52.512152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.3:32843 (local address 127.14.103.1:44025)
0812 02:34:52.512688 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0812 02:34:52.512705 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 02:34:52.512857 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:34:52.513308 (+ 451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:34:52.513312 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:34:52.513357 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:34:52.513644 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:34:52.513653 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:34:52.514918 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:34:52.514922 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:34:52.516721 (+ 1799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:34:52.516735 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:34:52.517745 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:34:52.517750 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:34:52.517949 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:52.518020 (+ 71us) client_negotiation.cc:769] Sending connection context
0812 02:34:52.518114 (+ 94us) client_negotiation.cc:240] Negotiation successful
0812 02:34:52.518209 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"mutex_wait_us":302,"spinlock_wait_cycles":24576,"thread_start_us":154,"threads_started":1}
I20260812 02:34:52.519654 30637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01c1cc2389af4f9198c7ec1c7a871ff0" candidate_uuid: "649a01114e0a46628578bff8843a050a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3909f7c94d654754a9d589bb0ae09e8a" is_pre_election: true
I20260812 02:34:52.520009 30637 raft_consensus.cc:2454] T 01c1cc2389af4f9198c7ec1c7a871ff0 P 3909f7c94d654754a9d589bb0ae09e8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 649a01114e0a46628578bff8843a050a in term 0.
I20260812 02:34:52.520663 30791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01c1cc2389af4f9198c7ec1c7a871ff0" candidate_uuid: "649a01114e0a46628578bff8843a050a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bfbbe1513aad4179abc52064d9250a9a" is_pre_election: true
I20260812 02:34:52.520733 30408 leader_election.cc:302] T 01c1cc2389af4f9198c7ec1c7a871ff0 P 649a01114e0a46628578bff8843a050a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3909f7c94d654754a9d589bb0ae09e8a, 649a01114e0a46628578bff8843a050a; no voters:
I20260812 02:34:52.520955 30791 raft_consensus.cc:2454] T 01c1cc2389af4f9198c7ec1c7a871ff0 P bfbbe1513aad4179abc52064d9250a9a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 649a01114e0a46628578bff8843a050a in term 0.
I20260812 02:34:52.521060 30871 raft_consensus.cc:2781] T 01c1cc2389af4f9198c7ec1c7a871ff0 P 649a01114e0a46628578bff8843a050a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 02:34:52.521173 30871 raft_consensus.cc:482] T 01c1cc2389af4f9198c7ec1c7a871ff0 P 649a01114e0a46628578bff8843a050a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 02:34:52.521235 30871 raft_consensus.cc:3037] T 01c1cc2389af4f9198c7ec1c7a871ff0 P 649a01114e0a46628578bff8843a050a [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:34:52.522393 30871 raft_consensus.cc:504] T 01c1cc2389af4f9198c7ec1c7a871ff0 P 649a01114e0a46628578bff8843a050a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfbbe1513aad4179abc52064d9250a9a" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 32843 } } peers { permanent_uuid: "649a01114e0a46628578bff8843a050a" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 45955 } } peers { permanent_uuid: "3909f7c94d654754a9d589bb0ae09e8a" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 46173 } }
I20260812 02:34:52.522768 30871 leader_election.cc:288] T 01c1cc2389af4f9198c7ec1c7a871ff0 P 649a01114e0a46628578bff8843a050a [CANDIDATE]: Term 1 election: Requested vote from peers bfbbe1513aad4179abc52064d9250a9a (127.14.103.3:32843), 3909f7c94d654754a9d589bb0ae09e8a (127.14.103.2:46173)
I20260812 02:34:52.523247 30791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01c1cc2389af4f9198c7ec1c7a871ff0" candidate_uuid: "649a01114e0a46628578bff8843a050a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bfbbe1513aad4179abc52064d9250a9a"
I20260812 02:34:52.523409 30791 raft_consensus.cc:3037] T 01c1cc2389af4f9198c7ec1c7a871ff0 P bfbbe1513aad4179abc52064d9250a9a [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:34:52.524534 30791 raft_consensus.cc:2454] T 01c1cc2389af4f9198c7ec1c7a871ff0 P bfbbe1513aad4179abc52064d9250a9a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 649a01114e0a46628578bff8843a050a in term 1.
I20260812 02:34:52.524824 30637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01c1cc2389af4f9198c7ec1c7a871ff0" candidate_uuid: "649a01114e0a46628578bff8843a050a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3909f7c94d654754a9d589bb0ae09e8a"
I20260812 02:34:52.524873 30407 leader_election.cc:302] T 01c1cc2389af4f9198c7ec1c7a871ff0 P 649a01114e0a46628578bff8843a050a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 649a01114e0a46628578bff8843a050a, bfbbe1513aad4179abc52064d9250a9a; no voters:
I20260812 02:34:52.524966 30637 raft_consensus.cc:3037] T 01c1cc2389af4f9198c7ec1c7a871ff0 P 3909f7c94d654754a9d589bb0ae09e8a [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:34:52.525111 30871 raft_consensus.cc:2781] T 01c1cc2389af4f9198c7ec1c7a871ff0 P 649a01114e0a46628578bff8843a050a [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:34:52.525384 30871 raft_consensus.cc:686] T 01c1cc2389af4f9198c7ec1c7a871ff0 P 649a01114e0a46628578bff8843a050a [term 1 LEADER]: Becoming Leader. State: Replica: 649a01114e0a46628578bff8843a050a, State: Running, Role: LEADER
I20260812 02:34:52.525769 30871 consensus_queue.cc:237] T 01c1cc2389af4f9198c7ec1c7a871ff0 P 649a01114e0a46628578bff8843a050a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfbbe1513aad4179abc52064d9250a9a" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 32843 } } peers { permanent_uuid: "649a01114e0a46628578bff8843a050a" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 45955 } } peers { permanent_uuid: "3909f7c94d654754a9d589bb0ae09e8a" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 46173 } }
I20260812 02:34:52.526358 30637 raft_consensus.cc:2454] T 01c1cc2389af4f9198c7ec1c7a871ff0 P 3909f7c94d654754a9d589bb0ae09e8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 649a01114e0a46628578bff8843a050a in term 1.
I20260812 02:34:52.529603 30311 catalog_manager.cc:5495] T 01c1cc2389af4f9198c7ec1c7a871ff0 P 649a01114e0a46628578bff8843a050a reported cstate change: term changed from 0 to 1, leader changed from <none> to 649a01114e0a46628578bff8843a050a (127.14.103.1). New cstate: current_term: 1 leader_uuid: "649a01114e0a46628578bff8843a050a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfbbe1513aad4179abc52064d9250a9a" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 32843 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "649a01114e0a46628578bff8843a050a" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 45955 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3909f7c94d654754a9d589bb0ae09e8a" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 46173 } health_report { overall_health: UNKNOWN } } }
I20260812 02:34:52.546921 30860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:52.543214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54394 (local address 127.14.103.1:45955)
0812 02:34:52.543365 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0812 02:34:52.543370 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:34:52.543387 (+ 17us) server_negotiation.cc:407] Connection header received
0812 02:34:52.543562 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:34:52.543564 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:34:52.543631 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:34:52.543738 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 02:34:52.544002 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:52.544515 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:52.545360 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:52.545587 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:52.545692 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:52.546022 (+ 330us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 02:34:52.546479 (+ 457us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 02:34:52.546615 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0812 02:34:52.546681 (+ 66us) server_negotiation.cc:299] Negotiation successful
0812 02:34:52.546756 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260812 02:34:52.553398 30791 raft_consensus.cc:1261] T 01c1cc2389af4f9198c7ec1c7a871ff0 P bfbbe1513aad4179abc52064d9250a9a [term 1 FOLLOWER]: Refusing update from remote peer 649a01114e0a46628578bff8843a050a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:34:52.553421 30637 raft_consensus.cc:1261] T 01c1cc2389af4f9198c7ec1c7a871ff0 P 3909f7c94d654754a9d589bb0ae09e8a [term 1 FOLLOWER]: Refusing update from remote peer 649a01114e0a46628578bff8843a050a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:34:52.554380 30871 consensus_queue.cc:1035] T 01c1cc2389af4f9198c7ec1c7a871ff0 P 649a01114e0a46628578bff8843a050a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3909f7c94d654754a9d589bb0ae09e8a" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 46173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:34:52.554637 30878 consensus_queue.cc:1035] T 01c1cc2389af4f9198c7ec1c7a871ff0 P 649a01114e0a46628578bff8843a050a [LEADER]: Connected to new peer: Peer: permanent_uuid: "bfbbe1513aad4179abc52064d9250a9a" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 32843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:34:52.566679 30883 mvcc.cc:205] Tried to move back new op lower bound from 7317512571090337792 to 7317512570988990464. Current Snapshot: MvccSnapshot[applied={T|T < 7317512571090337792 or (T in {7317512571090337792})}]
I20260812 02:34:52.574535 30311 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32992:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 02:34:52.574764 30311 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32992:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 02:34:52.577883 30311 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d92eab9524324b82b329fd272df49540: Sending DeleteTablet for 3 replicas of tablet 01c1cc2389af4f9198c7ec1c7a871ff0
I20260812 02:34:52.578909 30460 tablet_service.cc:1514] Processing DeleteTablet for tablet 01c1cc2389af4f9198c7ec1c7a871ff0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:34:52 UTC) from {username='kudu', principal='kudu/127.14.103.62@KRBTEST.COM'} at 127.0.0.1:54378
I20260812 02:34:52.578895 30616 tablet_service.cc:1514] Processing DeleteTablet for tablet 01c1cc2389af4f9198c7ec1c7a871ff0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:34:52 UTC) from {username='kudu', principal='kudu/127.14.103.62@KRBTEST.COM'} at 127.0.0.1:39610
I20260812 02:34:52.578850 30769 tablet_service.cc:1514] Processing DeleteTablet for tablet 01c1cc2389af4f9198c7ec1c7a871ff0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:34:52 UTC) from {username='kudu', principal='kudu/127.14.103.62@KRBTEST.COM'} at 127.0.0.1:58266
I20260812 02:34:52.579490 30892 tablet_replica.cc:330] T 01c1cc2389af4f9198c7ec1c7a871ff0 P 3909f7c94d654754a9d589bb0ae09e8a: stopping tablet replica
I20260812 02:34:52.579640 30893 tablet_replica.cc:330] T 01c1cc2389af4f9198c7ec1c7a871ff0 P bfbbe1513aad4179abc52064d9250a9a: stopping tablet replica
I20260812 02:34:52.579691 14748 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbep4M4/build/debug/bin/kudu with pid 30385
I20260812 02:34:52.579928 30893 raft_consensus.cc:2229] T 01c1cc2389af4f9198c7ec1c7a871ff0 P bfbbe1513aad4179abc52064d9250a9a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 02:34:52.579928 30892 raft_consensus.cc:2229] T 01c1cc2389af4f9198c7ec1c7a871ff0 P 3909f7c94d654754a9d589bb0ae09e8a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 02:34:52.580289 30893 raft_consensus.cc:2258] T 01c1cc2389af4f9198c7ec1c7a871ff0 P bfbbe1513aad4179abc52064d9250a9a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:34:52.582834 30892 raft_consensus.cc:2258] T 01c1cc2389af4f9198c7ec1c7a871ff0 P 3909f7c94d654754a9d589bb0ae09e8a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:34:52.585820 30893 ts_tablet_manager.cc:1905] T 01c1cc2389af4f9198c7ec1c7a871ff0 P bfbbe1513aad4179abc52064d9250a9a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:34:52.585820 30892 ts_tablet_manager.cc:1905] T 01c1cc2389af4f9198c7ec1c7a871ff0 P 3909f7c94d654754a9d589bb0ae09e8a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:34:52.588792 30892 ts_tablet_manager.cc:1918] T 01c1cc2389af4f9198c7ec1c7a871ff0 P 3909f7c94d654754a9d589bb0ae09e8a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 02:34:52.588792 30893 ts_tablet_manager.cc:1918] T 01c1cc2389af4f9198c7ec1c7a871ff0 P bfbbe1513aad4179abc52064d9250a9a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 02:34:52.588933 30893 log.cc:1196] T 01c1cc2389af4f9198c7ec1c7a871ff0 P bfbbe1513aad4179abc52064d9250a9a: Deleting WAL directory at /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/minicluster-data/ts-2/wal/wals/01c1cc2389af4f9198c7ec1c7a871ff0
I20260812 02:34:52.588933 30892 log.cc:1196] T 01c1cc2389af4f9198c7ec1c7a871ff0 P 3909f7c94d654754a9d589bb0ae09e8a: Deleting WAL directory at /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/minicluster-data/ts-1/wal/wals/01c1cc2389af4f9198c7ec1c7a871ff0
I20260812 02:34:52.589303 30892 ts_tablet_manager.cc:1939] T 01c1cc2389af4f9198c7ec1c7a871ff0 P 3909f7c94d654754a9d589bb0ae09e8a: Deleting consensus metadata
I20260812 02:34:52.589303 30893 ts_tablet_manager.cc:1939] T 01c1cc2389af4f9198c7ec1c7a871ff0 P bfbbe1513aad4179abc52064d9250a9a: Deleting consensus metadata
W20260812 02:34:52.591121 30297 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.103.1:45955 (error 108)
I20260812 02:34:52.591553 30296 catalog_manager.cc:4841] TS bfbbe1513aad4179abc52064d9250a9a (127.14.103.3:32843): tablet 01c1cc2389af4f9198c7ec1c7a871ff0 (table test-table [id=444bccb026034de8b4a56c055422dade]) successfully deleted
I20260812 02:34:52.591717 30297 catalog_manager.cc:4841] TS 3909f7c94d654754a9d589bb0ae09e8a (127.14.103.2:46173): tablet 01c1cc2389af4f9198c7ec1c7a871ff0 (table test-table [id=444bccb026034de8b4a56c055422dade]) successfully deleted
I20260812 02:34:52.592250 14748 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbep4M4/build/debug/bin/kudu with pid 30547
I20260812 02:34:52.592455 30861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:52.592060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.1:45955 (local address 127.0.0.1:54400)
0812 02:34:52.592229 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0812 02:34:52.592319 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.1:45955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260812 02:34:52.599979 30297 catalog_manager.cc:4584] TS 649a01114e0a46628578bff8843a050a (127.14.103.1:45955): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 01c1cc2389af4f9198c7ec1c7a871ff0: Network error: Client connection negotiation failed: client connection to 127.14.103.1:45955: connect: Connection refused (error 111)
I20260812 02:34:52.601181 14748 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbep4M4/build/debug/bin/kudu with pid 30696
I20260812 02:34:52.609994 14748 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbep4M4/build/debug/bin/kudu with pid 30277
I20260812 02:34:52.621138 14748 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbep4M4/build/debug/bin/kudu
/tmp/dist-test-taskbep4M4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.103.62:43307
--webserver_interface=127.14.103.62
--webserver_port=36911
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.62.keytab
--principal=kudu/127.14.103.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.103.20:36423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.103.62:43307
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:34:52.748092 30897 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:34:52.748430 30897 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:34:52.748476 30897 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:34:52.752826 30897 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:34:52.752914 30897 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:34:52.752934 30897 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:34:52.752954 30897 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:34:52.752969 30897 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 02:34:52.759318 30897 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.103.20:36423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.14.103.62:43307
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.62.keytab
--principal=kudu/127.14.103.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.103.62:43307
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.103.62
--webserver_port=36911
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30897
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:34:52.760501 30897 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:34:52.761541 30897 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:34:52.777215 30903 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:34:52.777462 30906 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:34:52.777529 30904 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:34:52.778008 30897 server_base.cc:1034] running on GCE node
I20260812 02:34:52.779012 30897 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:34:52.780486 30897 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:34:52.781730 30897 hybrid_clock.cc:648] HybridClock initialized: now 1786502092781668 us; error 85 us; skew 500 ppm
Aug 12 02:34:52 dist-test-slave-2w3w krb5kdc[30263](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502092, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:34:52.785214 30897 init.cc:377] Logged in from keytab as kudu/127.14.103.62@KRBTEST.COM (short username kudu)
I20260812 02:34:52.786334 30897 webserver.cc:460] Webserver started at http://127.14.103.62:36911/ using document root <none> and password file <none>
I20260812 02:34:52.787122 30897 fs_manager.cc:359] Metadata directory not provided
I20260812 02:34:52.787252 30897 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:34:52.792057 30897 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260812 02:34:52.794806 30914 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:34:52.796412 30897 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260812 02:34:52.796608 30897 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/minicluster-data/master-0/data,/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/minicluster-data/master-0/wal
uuid: "d92eab9524324b82b329fd272df49540"
format_stamp: "Formatted at 2026-08-12 02:34:51 on dist-test-slave-2w3w"
server_key: "664a157ae4b40297525e3039b779c47b"
server_key_iv: "295b86a832b5ed934d9001353bd656fc"
server_key_version: "encryptionkey@0"
I20260812 02:34:52.797336 30897 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:34:52.843988 30897 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:34:52.847803 30897 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:34:52.848074 30897 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:34:52.856442 30897 rpc_server.cc:307] RPC server started. Bound to: 127.14.103.62:43307
I20260812 02:34:52.856756 30976 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.103.62:43307 every 8 connection(s)
I20260812 02:34:52.857766 30897 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/minicluster-data/master-0/data/info.pb
I20260812 02:34:52.860534 14748 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbep4M4/build/debug/bin/kudu as pid 30897
I20260812 02:34:52.861058 14748 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbep4M4/build/debug/bin/kudu
/tmp/dist-test-taskbep4M4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.103.1:45955
--local_ip_for_outbound_sockets=127.14.103.1
--tserver_master_addrs=127.14.103.62:43307
--webserver_port=44913
--webserver_interface=127.14.103.1
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.1.keytab
--principal=kudu/127.14.103.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.103.20:36423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 02:34:52.864132 30977 sys_catalog.cc:263] Verifying existing consensus state
I20260812 02:34:52.866160 30977 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d92eab9524324b82b329fd272df49540: Bootstrap starting.
I20260812 02:34:52.885829 30977 log.cc:824] T 00000000000000000000000000000000 P d92eab9524324b82b329fd272df49540: Log is configured to *not* fsync() on all Append() calls
I20260812 02:34:52.896104 30977 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d92eab9524324b82b329fd272df49540: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 02:34:52.896570 30977 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d92eab9524324b82b329fd272df49540: Bootstrap complete.
I20260812 02:34:52.901088 30977 raft_consensus.cc:348] T 00000000000000000000000000000000 P d92eab9524324b82b329fd272df49540 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d92eab9524324b82b329fd272df49540" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 43307 } }
I20260812 02:34:52.901594 30977 raft_consensus.cc:729] T 00000000000000000000000000000000 P d92eab9524324b82b329fd272df49540 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d92eab9524324b82b329fd272df49540, State: Initialized, Role: FOLLOWER
I20260812 02:34:52.902163 30977 consensus_queue.cc:260] T 00000000000000000000000000000000 P d92eab9524324b82b329fd272df49540 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d92eab9524324b82b329fd272df49540" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 43307 } }
I20260812 02:34:52.902319 30977 raft_consensus.cc:388] T 00000000000000000000000000000000 P d92eab9524324b82b329fd272df49540 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 02:34:52.902411 30977 raft_consensus.cc:482] T 00000000000000000000000000000000 P d92eab9524324b82b329fd272df49540 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 02:34:52.902534 30977 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d92eab9524324b82b329fd272df49540 [term 1 FOLLOWER]: Advancing to term 2
I20260812 02:34:52.905233 30977 raft_consensus.cc:504] T 00000000000000000000000000000000 P d92eab9524324b82b329fd272df49540 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d92eab9524324b82b329fd272df49540" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 43307 } }
I20260812 02:34:52.905702 30977 leader_election.cc:302] T 00000000000000000000000000000000 P d92eab9524324b82b329fd272df49540 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d92eab9524324b82b329fd272df49540; no voters:
I20260812 02:34:52.906287 30977 leader_election.cc:288] T 00000000000000000000000000000000 P d92eab9524324b82b329fd272df49540 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260812 02:34:52.906433 30981 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d92eab9524324b82b329fd272df49540 [term 2 FOLLOWER]: Leader election won for term 2
I20260812 02:34:52.906695 30981 raft_consensus.cc:686] T 00000000000000000000000000000000 P d92eab9524324b82b329fd272df49540 [term 2 LEADER]: Becoming Leader. State: Replica: d92eab9524324b82b329fd272df49540, State: Running, Role: LEADER
I20260812 02:34:52.907089 30981 consensus_queue.cc:237] T 00000000000000000000000000000000 P d92eab9524324b82b329fd272df49540 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d92eab9524324b82b329fd272df49540" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 43307 } }
I20260812 02:34:52.907982 30977 sys_catalog.cc:564] T 00000000000000000000000000000000 P d92eab9524324b82b329fd272df49540 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:34:52.908710 30982 sys_catalog.cc:455] T 00000000000000000000000000000000 P d92eab9524324b82b329fd272df49540 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "d92eab9524324b82b329fd272df49540" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d92eab9524324b82b329fd272df49540" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 43307 } } }
I20260812 02:34:52.908851 30982 sys_catalog.cc:458] T 00000000000000000000000000000000 P d92eab9524324b82b329fd272df49540 [sys.catalog]: This master's current role is: LEADER
I20260812 02:34:52.909983 30983 sys_catalog.cc:455] T 00000000000000000000000000000000 P d92eab9524324b82b329fd272df49540 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d92eab9524324b82b329fd272df49540. Latest consensus state: current_term: 2 leader_uuid: "d92eab9524324b82b329fd272df49540" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d92eab9524324b82b329fd272df49540" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 43307 } } }
I20260812 02:34:52.910117 30983 sys_catalog.cc:458] T 00000000000000000000000000000000 P d92eab9524324b82b329fd272df49540 [sys.catalog]: This master's current role is: LEADER
I20260812 02:34:52.910336 30990 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 02:34:52.916504 30990 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 02:34:52.917249 30990 catalog_manager.cc:1259] Loaded cluster ID: fdb7ae612c83410bbce5c0d124a4977b
I20260812 02:34:52.917312 30990 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 02:34:52.918985 30990 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260812 02:34:52.923385 30990 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260812 02:34:52.929863 30990 catalog_manager.cc:1504] Loading token signing keys...
I20260812 02:34:52.930845 30990 catalog_manager.cc:5879] T 00000000000000000000000000000000 P d92eab9524324b82b329fd272df49540: Loaded TSK: 0
I20260812 02:34:52.932246 30990 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260812 02:34:52.995934 30979 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:34:52.996240 30979 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:34:52.996287 30979 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:34:53.000590 30979 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:34:53.000669 30979 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:34:53.000734 30979 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.103.1
I20260812 02:34:53.007709 30979 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.103.20:36423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.1.keytab
--principal=kudu/127.14.103.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.103.1:45955
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.103.1
--webserver_port=44913
--tserver_master_addrs=127.14.103.62:43307
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30979
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.103.1
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:34:53.009124 30979 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:34:53.010479 30979 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:34:53.022375 31007 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:34:53.022609 31008 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:34:53.022763 31010 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:34:53.022624 30979 server_base.cc:1034] running on GCE node
I20260812 02:34:53.023416 30979 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:34:53.024283 30979 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:34:53.025481 30979 hybrid_clock.cc:648] HybridClock initialized: now 1786502093025458 us; error 39 us; skew 500 ppm
Aug 12 02:34:53 dist-test-slave-2w3w krb5kdc[30263](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502093, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:34:53.028930 30979 init.cc:377] Logged in from keytab as kudu/127.14.103.1@KRBTEST.COM (short username kudu)
I20260812 02:34:53.030095 30979 webserver.cc:460] Webserver started at http://127.14.103.1:44913/ using document root <none> and password file <none>
I20260812 02:34:53.030726 30979 fs_manager.cc:359] Metadata directory not provided
I20260812 02:34:53.030844 30979 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:34:53.035627 30979 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260812 02:34:53.038231 31018 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:34:53.039803 30979 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 02:34:53.040001 30979 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/minicluster-data/ts-0/wal
uuid: "649a01114e0a46628578bff8843a050a"
format_stamp: "Formatted at 2026-08-12 02:34:51 on dist-test-slave-2w3w"
server_key: "635812a0cc12b707e20ed3264aacc3ea"
server_key_iv: "37d30bf5c49236f63ef0991a57da38d0"
server_key_version: "encryptionkey@0"
I20260812 02:34:53.040614 30979 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:34:53.069510 30979 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:34:53.073470 30979 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:34:53.073719 30979 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:34:53.074404 30979 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:34:53.075724 31026 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260812 02:34:53.080729 30979 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260812 02:34:53.080859 30979 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.001s
I20260812 02:34:53.081035 30979 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260812 02:34:53.083029 30979 ts_tablet_manager.cc:610] Registered 1 tablets
I20260812 02:34:53.083118 30979 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.002s
I20260812 02:34:53.083443 31026 tablet_bootstrap.cc:492] T 01c1cc2389af4f9198c7ec1c7a871ff0 P 649a01114e0a46628578bff8843a050a: Bootstrap starting.
I20260812 02:34:53.094393 30979 rpc_server.cc:307] RPC server started. Bound to: 127.14.103.1:45955
I20260812 02:34:53.094410 31148 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.103.1:45955 every 8 connection(s)
I20260812 02:34:53.095561 30979 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/minicluster-data/ts-0/data/info.pb
I20260812 02:34:53.100889 14748 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbep4M4/build/debug/bin/kudu as pid 30979
I20260812 02:34:53.101553 14748 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbep4M4/build/debug/bin/kudu
/tmp/dist-test-taskbep4M4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.103.2:46173
--local_ip_for_outbound_sockets=127.14.103.2
--tserver_master_addrs=127.14.103.62:43307
--webserver_port=34039
--webserver_interface=127.14.103.2
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.2.keytab
--principal=kudu/127.14.103.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.103.20:36423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 02:34:53.104743 31026 log.cc:824] T 01c1cc2389af4f9198c7ec1c7a871ff0 P 649a01114e0a46628578bff8843a050a: Log is configured to *not* fsync() on all Append() calls
Aug 12 02:34:53 dist-test-slave-2w3w krb5kdc[30263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502093, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.1@KRBTEST.COM for kudu/127.14.103.62@KRBTEST.COM
I20260812 02:34:53.116008 31153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:53.097667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:43307 (local address 127.14.103.1:33587)
0812 02:34:53.098194 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0812 02:34:53.098253 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0812 02:34:53.099183 (+ 930us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:34:53.100822 (+ 1639us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:34:53.100868 (+ 46us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:34:53.101536 (+ 668us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:34:53.102311 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:34:53.102339 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:34:53.104703 (+ 2364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:34:53.104710 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:34:53.106144 (+ 1434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:34:53.106156 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:34:53.107014 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:34:53.107019 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:34:53.107204 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:53.107984 (+ 780us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:34:53.108002 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:34:53.109995 (+ 1993us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:34:53.112469 (+ 2474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:34:53.112475 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:34:53.112488 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:34:53.112776 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:34:53.113233 (+ 457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:34:53.113237 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:34:53.113241 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:34:53.113417 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:34:53.114103 (+ 686us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:34:53.114109 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:34:53.114370 (+ 261us) client_negotiation.cc:769] Sending connection context
0812 02:34:53.114582 (+ 212us) client_negotiation.cc:240] Negotiation successful
0812 02:34:53.115015 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":166,"threads_started":1}
I20260812 02:34:53.115991 31156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:53.097716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.1:33587 (local address 127.14.103.62:43307)
0812 02:34:53.098287 (+ 571us) server_negotiation.cc:206] Beginning negotiation
0812 02:34:53.098306 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0812 02:34:53.098369 (+ 63us) server_negotiation.cc:407] Connection header received
0812 02:34:53.099640 (+ 1271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:34:53.099656 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:34:53.100099 (+ 443us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:34:53.100578 (+ 479us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:34:53.102554 (+ 1976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:53.104439 (+ 1885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:53.106298 (+ 1859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:53.106772 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:53.107068 (+ 296us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:53.110265 (+ 3197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:34:53.110272 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:34:53.110285 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:34:53.110356 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:34:53.112310 (+ 1954us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:34:53.112910 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:34:53.112916 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:34:53.112926 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:34:53.113054 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:34:53.113524 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:34:53.113529 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:34:53.113531 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:34:53.113977 (+ 446us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:34:53.114133 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0812 02:34:53.114701 (+ 568us) server_negotiation.cc:299] Negotiation successful
0812 02:34:53.115036 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":420,"thread_start_us":158,"threads_started":1}
I20260812 02:34:53.116470 31026 tablet_bootstrap.cc:492] T 01c1cc2389af4f9198c7ec1c7a871ff0 P 649a01114e0a46628578bff8843a050a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 02:34:53.117015 31026 tablet_bootstrap.cc:492] T 01c1cc2389af4f9198c7ec1c7a871ff0 P 649a01114e0a46628578bff8843a050a: Bootstrap complete.
I20260812 02:34:53.118662 31026 ts_tablet_manager.cc:1397] T 01c1cc2389af4f9198c7ec1c7a871ff0 P 649a01114e0a46628578bff8843a050a: Time spent bootstrapping tablet: real 0.035s user 0.018s sys 0.014s
I20260812 02:34:53.118868 31150 heartbeater.cc:347] Connected to a master server at 127.14.103.62:43307
I20260812 02:34:53.119134 31150 heartbeater.cc:464] Registering TS with master...
I20260812 02:34:53.119668 31150 heartbeater.cc:511] Master 127.14.103.62:43307 requested a full tablet report, sending...
I20260812 02:34:53.122117 30930 ts_manager.cc:194] Registered new tserver with Master: 649a01114e0a46628578bff8843a050a (127.14.103.1:45955)
I20260812 02:34:53.121873 31026 raft_consensus.cc:348] T 01c1cc2389af4f9198c7ec1c7a871ff0 P 649a01114e0a46628578bff8843a050a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfbbe1513aad4179abc52064d9250a9a" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 32843 } } peers { permanent_uuid: "649a01114e0a46628578bff8843a050a" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 45955 } } peers { permanent_uuid: "3909f7c94d654754a9d589bb0ae09e8a" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 46173 } }
I20260812 02:34:53.122442 31026 raft_consensus.cc:729] T 01c1cc2389af4f9198c7ec1c7a871ff0 P 649a01114e0a46628578bff8843a050a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 649a01114e0a46628578bff8843a050a, State: Initialized, Role: FOLLOWER
I20260812 02:34:53.122939 31026 consensus_queue.cc:260] T 01c1cc2389af4f9198c7ec1c7a871ff0 P 649a01114e0a46628578bff8843a050a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfbbe1513aad4179abc52064d9250a9a" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 32843 } } peers { permanent_uuid: "649a01114e0a46628578bff8843a050a" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 45955 } } peers { permanent_uuid: "3909f7c94d654754a9d589bb0ae09e8a" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 46173 } }
I20260812 02:34:53.124071 31026 ts_tablet_manager.cc:1428] T 01c1cc2389af4f9198c7ec1c7a871ff0 P 649a01114e0a46628578bff8843a050a: Time spent starting tablet: real 0.005s user 0.006s sys 0.003s
I20260812 02:34:53.126084 30930 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.103.1@KRBTEST.COM'} at 127.14.103.1:33587
I20260812 02:34:53.128576 31150 heartbeater.cc:503] Master 127.14.103.62:43307 was elected leader, sending a full tablet report...
Aug 12 02:34:53 dist-test-slave-2w3w krb5kdc[30263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502092, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.62@KRBTEST.COM for kudu/127.14.103.1@KRBTEST.COM
I20260812 02:34:53.138739 31162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:53.125508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54408 (local address 127.14.103.1:45955)
0812 02:34:53.126560 (+ 1052us) server_negotiation.cc:206] Beginning negotiation
0812 02:34:53.126572 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 02:34:53.126595 (+ 23us) server_negotiation.cc:407] Connection header received
0812 02:34:53.126667 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:34:53.126672 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:34:53.126879 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:34:53.127050 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:34:53.129488 (+ 2438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:53.130532 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:53.131816 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:53.132174 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:53.132266 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:53.135002 (+ 2736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:34:53.135014 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:34:53.135030 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:34:53.135054 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:34:53.136690 (+ 1636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:34:53.137223 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:34:53.137228 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:34:53.137229 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:34:53.137311 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:34:53.137609 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:34:53.137612 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:34:53.137614 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:34:53.137897 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:34:53.138034 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0812 02:34:53.138455 (+ 421us) server_negotiation.cc:299] Negotiation successful
0812 02:34:53.138606 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":954,"thread_start_us":118,"threads_started":1}
I20260812 02:34:53.138767 31161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:53.125382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.1:45955 (local address 127.0.0.1:54408)
0812 02:34:53.126060 (+ 678us) negotiation.cc:107] Waiting for socket to connect
0812 02:34:53.126118 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0812 02:34:53.126271 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:34:53.127058 (+ 787us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:34:53.127065 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:34:53.129159 (+ 2094us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:34:53.129330 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:34:53.129342 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:34:53.130655 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:34:53.130659 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:34:53.131669 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:34:53.131682 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:34:53.132300 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:34:53.132304 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:34:53.132431 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:53.133007 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:34:53.133023 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:34:53.134804 (+ 1781us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:34:53.136820 (+ 2016us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:34:53.136838 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:34:53.136844 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:34:53.137096 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:34:53.137415 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:34:53.137419 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:34:53.137420 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:34:53.137511 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:34:53.138049 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:34:53.138057 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:34:53.138197 (+ 140us) client_negotiation.cc:769] Sending connection context
0812 02:34:53.138367 (+ 170us) client_negotiation.cc:240] Negotiation successful
0812 02:34:53.138585 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":127,"threads_started":1}
I20260812 02:34:53.140030 31073 tablet_service.cc:1514] Processing DeleteTablet for tablet 01c1cc2389af4f9198c7ec1c7a871ff0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:34:52 UTC) from {username='kudu', principal='kudu/127.14.103.62@KRBTEST.COM'} at 127.0.0.1:54408
I20260812 02:34:53.140936 31163 tablet_replica.cc:330] stopping tablet replica
I20260812 02:34:53.141243 31163 raft_consensus.cc:2229] T 01c1cc2389af4f9198c7ec1c7a871ff0 P 649a01114e0a46628578bff8843a050a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 02:34:53.141501 31163 raft_consensus.cc:2258] T 01c1cc2389af4f9198c7ec1c7a871ff0 P 649a01114e0a46628578bff8843a050a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:34:53.142474 31163 ts_tablet_manager.cc:1905] T 01c1cc2389af4f9198c7ec1c7a871ff0 P 649a01114e0a46628578bff8843a050a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:34:53.146364 31163 ts_tablet_manager.cc:1918] T 01c1cc2389af4f9198c7ec1c7a871ff0 P 649a01114e0a46628578bff8843a050a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 02:34:53.146464 31163 log.cc:1196] T 01c1cc2389af4f9198c7ec1c7a871ff0 P 649a01114e0a46628578bff8843a050a: Deleting WAL directory at /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/minicluster-data/ts-0/wal/wals/01c1cc2389af4f9198c7ec1c7a871ff0
I20260812 02:34:53.146757 31163 ts_tablet_manager.cc:1939] T 01c1cc2389af4f9198c7ec1c7a871ff0 P 649a01114e0a46628578bff8843a050a: Deleting consensus metadata
I20260812 02:34:53.148793 30917 catalog_manager.cc:4841] TS 649a01114e0a46628578bff8843a050a (127.14.103.1:45955): tablet 01c1cc2389af4f9198c7ec1c7a871ff0 (table test-table [id=444bccb026034de8b4a56c055422dade]) successfully deleted
W20260812 02:34:53.229197 31157 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:34:53.229589 31157 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:34:53.229701 31157 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:34:53.233955 31157 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:34:53.234064 31157 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:34:53.234160 31157 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.103.2
I20260812 02:34:53.239789 31157 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.103.20:36423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.2.keytab
--principal=kudu/127.14.103.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.103.2:46173
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.103.2
--webserver_port=34039
--tserver_master_addrs=127.14.103.62:43307
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31157
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.103.2
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:34:53.240970 31157 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:34:53.242031 31157 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:34:53.252252 31171 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:34:53.252269 31169 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:34:53.252244 31168 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:34:53.252503 31157 server_base.cc:1034] running on GCE node
I20260812 02:34:53.253070 31157 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:34:53.253819 31157 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:34:53.255038 31157 hybrid_clock.cc:648] HybridClock initialized: now 1786502093255005 us; error 50 us; skew 500 ppm
Aug 12 02:34:53 dist-test-slave-2w3w krb5kdc[30263](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502093, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:34:53.257953 31157 init.cc:377] Logged in from keytab as kudu/127.14.103.2@KRBTEST.COM (short username kudu)
I20260812 02:34:53.258881 31157 webserver.cc:460] Webserver started at http://127.14.103.2:34039/ using document root <none> and password file <none>
I20260812 02:34:53.259538 31157 fs_manager.cc:359] Metadata directory not provided
I20260812 02:34:53.259657 31157 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:34:53.263877 31157 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260812 02:34:53.266284 31181 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:34:53.267728 31157 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260812 02:34:53.267872 31157 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/minicluster-data/ts-1/wal
uuid: "3909f7c94d654754a9d589bb0ae09e8a"
format_stamp: "Formatted at 2026-08-12 02:34:52 on dist-test-slave-2w3w"
server_key: "2f5c8300d53314fffe9ae0ed32f4ad3e"
server_key_iv: "d2bd2e696ab65ca38afef510784b98ef"
server_key_version: "encryptionkey@0"
I20260812 02:34:53.268476 31157 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:34:53.312343 31157 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:34:53.315847 31157 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:34:53.316074 31157 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:34:53.316699 31157 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:34:53.317585 31157 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:34:53.317662 31157 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:34:53.317734 31157 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:34:53.317785 31157 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:34:53.327425 31157 rpc_server.cc:307] RPC server started. Bound to: 127.14.103.2:46173
I20260812 02:34:53.327450 31309 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.103.2:46173 every 8 connection(s)
I20260812 02:34:53.328469 31157 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/minicluster-data/ts-1/data/info.pb
I20260812 02:34:53.331871 14748 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbep4M4/build/debug/bin/kudu as pid 31157
I20260812 02:34:53.332324 14748 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbep4M4/build/debug/bin/kudu
/tmp/dist-test-taskbep4M4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.103.3:32843
--local_ip_for_outbound_sockets=127.14.103.3
--tserver_master_addrs=127.14.103.62:43307
--webserver_port=33687
--webserver_interface=127.14.103.3
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.3.keytab
--principal=kudu/127.14.103.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.103.20:36423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 02:34:53 dist-test-slave-2w3w krb5kdc[30263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502093, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.2@KRBTEST.COM for kudu/127.14.103.62@KRBTEST.COM
I20260812 02:34:53.343538 31156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:53.330468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.2:35719 (local address 127.14.103.62:43307)
0812 02:34:53.330646 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0812 02:34:53.330650 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:34:53.331608 (+ 958us) server_negotiation.cc:407] Connection header received
0812 02:34:53.333174 (+ 1566us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:34:53.333178 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:34:53.333243 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:34:53.333324 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:34:53.334591 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:53.335271 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:53.336440 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:53.336682 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:53.336791 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:53.339476 (+ 2685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:34:53.339479 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:34:53.339481 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:34:53.339508 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:34:53.341131 (+ 1623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:34:53.341747 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:34:53.341755 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:34:53.341757 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:34:53.341842 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:34:53.342245 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:34:53.342248 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:34:53.342250 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:34:53.342464 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:34:53.342577 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0812 02:34:53.343187 (+ 610us) server_negotiation.cc:299] Negotiation successful
0812 02:34:53.343341 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260812 02:34:53.344472 31313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:53.330822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:43307 (local address 127.14.103.2:35719)
0812 02:34:53.331417 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0812 02:34:53.331463 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0812 02:34:53.332916 (+ 1453us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:34:53.333522 (+ 606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:34:53.333529 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:34:53.333916 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:34:53.334425 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:34:53.334438 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:34:53.335404 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:34:53.335412 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:34:53.336304 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:34:53.336314 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:34:53.336850 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:34:53.336855 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:34:53.337033 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:53.337688 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:34:53.337704 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:34:53.339281 (+ 1577us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:34:53.341288 (+ 2007us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:34:53.341294 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:34:53.341313 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:34:53.341624 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:34:53.341945 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:34:53.341948 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:34:53.341950 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:34:53.342122 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:34:53.342575 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:34:53.342582 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:34:53.342868 (+ 286us) client_negotiation.cc:769] Sending connection context
0812 02:34:53.343129 (+ 261us) client_negotiation.cc:240] Negotiation successful
0812 02:34:53.343501 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":185,"threads_started":1}
I20260812 02:34:53.345788 31310 heartbeater.cc:347] Connected to a master server at 127.14.103.62:43307
I20260812 02:34:53.346042 31310 heartbeater.cc:464] Registering TS with master...
I20260812 02:34:53.346604 31310 heartbeater.cc:511] Master 127.14.103.62:43307 requested a full tablet report, sending...
I20260812 02:34:53.347766 30930 ts_manager.cc:194] Registered new tserver with Master: 3909f7c94d654754a9d589bb0ae09e8a (127.14.103.2:46173)
I20260812 02:34:53.348390 30930 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.103.2@KRBTEST.COM'} at 127.14.103.2:35719
W20260812 02:34:53.453243 31314 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:34:53.453611 31314 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:34:53.453698 31314 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:34:53.458025 31314 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:34:53.458134 31314 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:34:53.458227 31314 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.103.3
I20260812 02:34:53.464102 31314 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.103.20:36423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.3.keytab
--principal=kudu/127.14.103.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.103.3:32843
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.103.3
--webserver_port=33687
--tserver_master_addrs=127.14.103.62:43307
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31314
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.103.3
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:34:53.465276 31314 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:34:53.466355 31314 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:34:53.476639 31322 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:34:53.476723 31314 server_base.cc:1034] running on GCE node
W20260812 02:34:53.476639 31323 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:34:53.476658 31325 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:34:53.477385 31314 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:34:53.478129 31314 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:34:53.479326 31314 hybrid_clock.cc:648] HybridClock initialized: now 1786502093479309 us; error 32 us; skew 500 ppm
Aug 12 02:34:53 dist-test-slave-2w3w krb5kdc[30263](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502093, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:34:53.482247 31314 init.cc:377] Logged in from keytab as kudu/127.14.103.3@KRBTEST.COM (short username kudu)
I20260812 02:34:53.483251 31314 webserver.cc:460] Webserver started at http://127.14.103.3:33687/ using document root <none> and password file <none>
I20260812 02:34:53.483837 31314 fs_manager.cc:359] Metadata directory not provided
I20260812 02:34:53.483961 31314 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:34:53.488256 31314 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260812 02:34:53.490752 31332 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:34:53.492327 31314 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260812 02:34:53.492507 31314 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/minicluster-data/ts-2/wal
uuid: "bfbbe1513aad4179abc52064d9250a9a"
format_stamp: "Formatted at 2026-08-12 02:34:52 on dist-test-slave-2w3w"
server_key: "5a89d8ac702231998b7f2134835d7d8c"
server_key_iv: "ab713679a6525a8ff3e27f67fb2b5a2b"
server_key_version: "encryptionkey@0"
I20260812 02:34:53.493144 31314 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:34:53.505837 31314 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:34:53.509552 31314 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:34:53.509763 31314 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:34:53.510454 31314 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:34:53.511384 31314 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:34:53.511449 31314 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:34:53.511518 31314 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:34:53.511653 31314 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:34:53.521279 31314 rpc_server.cc:307] RPC server started. Bound to: 127.14.103.3:32843
I20260812 02:34:53.521319 31457 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.103.3:32843 every 8 connection(s)
I20260812 02:34:53.522459 31314 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/minicluster-data/ts-2/data/info.pb
I20260812 02:34:53.530591 14748 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbep4M4/build/debug/bin/kudu as pid 31314
Aug 12 02:34:53 dist-test-slave-2w3w krb5kdc[30263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502093, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.3@KRBTEST.COM for kudu/127.14.103.62@KRBTEST.COM
I20260812 02:34:53.539040 31156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:53.524418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.3:40929 (local address 127.14.103.62:43307)
0812 02:34:53.524556 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0812 02:34:53.524559 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 02:34:53.525442 (+ 883us) server_negotiation.cc:407] Connection header received
0812 02:34:53.527234 (+ 1792us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:34:53.527238 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:34:53.527320 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:34:53.527433 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:34:53.528782 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:53.529391 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:53.530680 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:53.531073 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:53.531221 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:53.534760 (+ 3539us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:34:53.534764 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:34:53.534766 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:34:53.534791 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:34:53.536559 (+ 1768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:34:53.537224 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:34:53.537229 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:34:53.537230 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:34:53.537321 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:34:53.537748 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:34:53.537752 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:34:53.537754 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:34:53.537947 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:34:53.538059 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0812 02:34:53.538596 (+ 537us) server_negotiation.cc:299] Negotiation successful
0812 02:34:53.538831 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260812 02:34:53.540030 31460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:53.524732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:43307 (local address 127.14.103.3:40929)
0812 02:34:53.525266 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0812 02:34:53.525312 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0812 02:34:53.526909 (+ 1597us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:34:53.527641 (+ 732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:34:53.527649 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:34:53.528084 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:34:53.528617 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:34:53.528631 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:34:53.529537 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:34:53.529541 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:34:53.530536 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:34:53.530547 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:34:53.531303 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:34:53.531311 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:34:53.531535 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:53.532406 (+ 871us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:34:53.532433 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:34:53.534543 (+ 2110us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:34:53.536686 (+ 2143us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:34:53.536696 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:34:53.536718 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:34:53.537130 (+ 412us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:34:53.537414 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:34:53.537416 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:34:53.537417 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:34:53.537612 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:34:53.538058 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:34:53.538066 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:34:53.538348 (+ 282us) client_negotiation.cc:769] Sending connection context
0812 02:34:53.538550 (+ 202us) client_negotiation.cc:240] Negotiation successful
0812 02:34:53.538935 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":127,"threads_started":1}
I20260812 02:34:53.540635 31462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:53.531495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32994 (local address 127.14.103.62:43307)
0812 02:34:53.531811 (+ 316us) server_negotiation.cc:206] Beginning negotiation
0812 02:34:53.531815 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:34:53.531947 (+ 132us) server_negotiation.cc:407] Connection header received
0812 02:34:53.532082 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:34:53.532086 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:34:53.532155 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:34:53.532263 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:34:53.533326 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:53.534195 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:53.535292 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:53.535669 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:53.535777 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:53.537207 (+ 1430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:34:53.537213 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:34:53.537216 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:34:53.537262 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:34:53.538854 (+ 1592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:34:53.539367 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:34:53.539371 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:34:53.539373 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:34:53.539461 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:34:53.539745 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:34:53.539748 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:34:53.539750 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:34:53.539934 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:34:53.540173 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0812 02:34:53.540309 (+ 136us) server_negotiation.cc:299] Negotiation successful
0812 02:34:53.540452 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":234,"spinlock_wait_cycles":6400,"thread_start_us":118,"threads_started":1}
I20260812 02:34:53.541412 31458 heartbeater.cc:347] Connected to a master server at 127.14.103.62:43307
I20260812 02:34:53.541707 31458 heartbeater.cc:464] Registering TS with master...
I20260812 02:34:53.542253 31458 heartbeater.cc:511] Master 127.14.103.62:43307 requested a full tablet report, sending...
I20260812 02:34:53.543393 30930 ts_manager.cc:194] Registered new tserver with Master: bfbbe1513aad4179abc52064d9250a9a (127.14.103.3:32843)
I20260812 02:34:53.543989 30930 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.103.3@KRBTEST.COM'} at 127.14.103.3:40929
I20260812 02:34:53.554945 14748 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 02:34:53.559970 31462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:53.555716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33000 (local address 127.14.103.62:43307)
0812 02:34:53.555855 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0812 02:34:53.555858 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 02:34:53.556084 (+ 226us) server_negotiation.cc:407] Connection header received
0812 02:34:53.556224 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:34:53.556246 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:34:53.556360 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:34:53.556463 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 02:34:53.556688 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:53.557246 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:53.558221 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:53.558475 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:53.558569 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:53.559083 (+ 514us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 02:34:53.559544 (+ 461us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 02:34:53.559648 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0812 02:34:53.559772 (+ 124us) server_negotiation.cc:299] Negotiation successful
0812 02:34:53.559848 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260812 02:34:53.560784 30930 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:33000:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 02:34:53.564225 30930 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 02:34:53.572867 31073 tablet_service.cc:1467] Processing CreateTablet for tablet ca63dc3c7cdf4b0781fbfc18c270a3a5 (DEFAULT_TABLE table=test-table [id=a9e33cc803e44b969a8585e4f784818a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:34:53.573621 31073 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca63dc3c7cdf4b0781fbfc18c270a3a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:34:53.576793 31026 tablet_bootstrap.cc:492] T ca63dc3c7cdf4b0781fbfc18c270a3a5 P 649a01114e0a46628578bff8843a050a: Bootstrap starting.
I20260812 02:34:53.577790 31161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:53.571570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.2:46173 (local address 127.0.0.1:39614)
0812 02:34:53.571687 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0812 02:34:53.571706 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 02:34:53.571786 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:34:53.573109 (+ 1323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:34:53.573112 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:34:53.573131 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:34:53.573257 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:34:53.573265 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:34:53.575117 (+ 1852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:34:53.575121 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:34:53.576121 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:34:53.576132 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:34:53.577273 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:34:53.577277 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:34:53.577395 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:53.577474 (+ 79us) client_negotiation.cc:769] Sending connection context
0812 02:34:53.577534 (+ 60us) client_negotiation.cc:240] Negotiation successful
0812 02:34:53.577633 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38,"mutex_wait_us":94}
I20260812 02:34:53.577877 31026 tablet_bootstrap.cc:654] T ca63dc3c7cdf4b0781fbfc18c270a3a5 P 649a01114e0a46628578bff8843a050a: Neither blocks nor log segments found. Creating new log.
I20260812 02:34:53.578715 31470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:53.571988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39614 (local address 127.14.103.2:46173)
0812 02:34:53.572571 (+ 583us) server_negotiation.cc:206] Beginning negotiation
0812 02:34:53.572585 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0812 02:34:53.572610 (+ 25us) server_negotiation.cc:407] Connection header received
0812 02:34:53.572693 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:34:53.572699 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:34:53.572935 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:34:53.573293 (+ 358us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:34:53.573481 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:53.574940 (+ 1459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:53.576262 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:53.577140 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:53.578105 (+ 965us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:53.578320 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0812 02:34:53.578410 (+ 90us) server_negotiation.cc:299] Negotiation successful
0812 02:34:53.578522 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":471,"thread_start_us":163,"threads_started":1}
I20260812 02:34:53.578915 31468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:53.571953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.3:32843 (local address 127.0.0.1:58272)
0812 02:34:53.572647 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0812 02:34:53.572663 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 02:34:53.572795 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:34:53.573605 (+ 810us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:34:53.573609 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:34:53.573628 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:34:53.573772 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:34:53.573780 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:34:53.575167 (+ 1387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:34:53.575171 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:34:53.576410 (+ 1239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:34:53.576423 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:34:53.578406 (+ 1983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:34:53.578410 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:34:53.578512 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:53.578537 (+ 25us) client_negotiation.cc:769] Sending connection context
0812 02:34:53.578639 (+ 102us) client_negotiation.cc:240] Negotiation successful
0812 02:34:53.578728 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":561,"mutex_wait_us":33,"thread_start_us":125,"threads_started":1}
I20260812 02:34:53.579342 31472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:53.572308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58272 (local address 127.14.103.3:32843)
0812 02:34:53.572934 (+ 626us) server_negotiation.cc:206] Beginning negotiation
0812 02:34:53.572944 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 02:34:53.572963 (+ 19us) server_negotiation.cc:407] Connection header received
0812 02:34:53.573024 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:34:53.573028 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:34:53.573224 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:34:53.573387 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:34:53.573898 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:53.574951 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:53.576557 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:53.577920 (+ 1363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:53.578053 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:53.578320 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0812 02:34:53.578704 (+ 384us) server_negotiation.cc:299] Negotiation successful
0812 02:34:53.579188 (+ 484us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":478,"thread_start_us":238,"threads_started":1}
I20260812 02:34:53.582443 31384 tablet_service.cc:1467] Processing CreateTablet for tablet ca63dc3c7cdf4b0781fbfc18c270a3a5 (DEFAULT_TABLE table=test-table [id=a9e33cc803e44b969a8585e4f784818a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:34:53.582443 31235 tablet_service.cc:1467] Processing CreateTablet for tablet ca63dc3c7cdf4b0781fbfc18c270a3a5 (DEFAULT_TABLE table=test-table [id=a9e33cc803e44b969a8585e4f784818a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:34:53.583735 31235 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca63dc3c7cdf4b0781fbfc18c270a3a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:34:53.583740 31384 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca63dc3c7cdf4b0781fbfc18c270a3a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:34:53.584138 31026 tablet_bootstrap.cc:492] T ca63dc3c7cdf4b0781fbfc18c270a3a5 P 649a01114e0a46628578bff8843a050a: No bootstrap required, opened a new log
I20260812 02:34:53.584272 31026 ts_tablet_manager.cc:1397] T ca63dc3c7cdf4b0781fbfc18c270a3a5 P 649a01114e0a46628578bff8843a050a: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.001s
I20260812 02:34:53.584792 31026 raft_consensus.cc:348] T ca63dc3c7cdf4b0781fbfc18c270a3a5 P 649a01114e0a46628578bff8843a050a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3909f7c94d654754a9d589bb0ae09e8a" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 46173 } } peers { permanent_uuid: "bfbbe1513aad4179abc52064d9250a9a" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 32843 } } peers { permanent_uuid: "649a01114e0a46628578bff8843a050a" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 45955 } }
I20260812 02:34:53.584932 31026 raft_consensus.cc:374] T ca63dc3c7cdf4b0781fbfc18c270a3a5 P 649a01114e0a46628578bff8843a050a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:34:53.584985 31026 raft_consensus.cc:729] T ca63dc3c7cdf4b0781fbfc18c270a3a5 P 649a01114e0a46628578bff8843a050a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 649a01114e0a46628578bff8843a050a, State: Initialized, Role: FOLLOWER
I20260812 02:34:53.585112 31026 consensus_queue.cc:260] T ca63dc3c7cdf4b0781fbfc18c270a3a5 P 649a01114e0a46628578bff8843a050a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3909f7c94d654754a9d589bb0ae09e8a" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 46173 } } peers { permanent_uuid: "bfbbe1513aad4179abc52064d9250a9a" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 32843 } } peers { permanent_uuid: "649a01114e0a46628578bff8843a050a" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 45955 } }
I20260812 02:34:53.585775 31026 ts_tablet_manager.cc:1428] T ca63dc3c7cdf4b0781fbfc18c270a3a5 P 649a01114e0a46628578bff8843a050a: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20260812 02:34:53.593133 31475 tablet_bootstrap.cc:492] T ca63dc3c7cdf4b0781fbfc18c270a3a5 P bfbbe1513aad4179abc52064d9250a9a: Bootstrap starting.
I20260812 02:34:53.593713 31476 tablet_bootstrap.cc:492] T ca63dc3c7cdf4b0781fbfc18c270a3a5 P 3909f7c94d654754a9d589bb0ae09e8a: Bootstrap starting.
I20260812 02:34:53.596033 31475 tablet_bootstrap.cc:654] T ca63dc3c7cdf4b0781fbfc18c270a3a5 P bfbbe1513aad4179abc52064d9250a9a: Neither blocks nor log segments found. Creating new log.
I20260812 02:34:53.596546 31476 tablet_bootstrap.cc:654] T ca63dc3c7cdf4b0781fbfc18c270a3a5 P 3909f7c94d654754a9d589bb0ae09e8a: Neither blocks nor log segments found. Creating new log.
W20260812 02:34:53.596969 31151 tablet.cc:2369] T ca63dc3c7cdf4b0781fbfc18c270a3a5 P 649a01114e0a46628578bff8843a050a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 02:34:53.598039 31475 log.cc:824] T ca63dc3c7cdf4b0781fbfc18c270a3a5 P bfbbe1513aad4179abc52064d9250a9a: Log is configured to *not* fsync() on all Append() calls
I20260812 02:34:53.598304 31476 log.cc:824] T ca63dc3c7cdf4b0781fbfc18c270a3a5 P 3909f7c94d654754a9d589bb0ae09e8a: Log is configured to *not* fsync() on all Append() calls
I20260812 02:34:53.600415 31476 tablet_bootstrap.cc:492] T ca63dc3c7cdf4b0781fbfc18c270a3a5 P 3909f7c94d654754a9d589bb0ae09e8a: No bootstrap required, opened a new log
I20260812 02:34:53.600431 31475 tablet_bootstrap.cc:492] T ca63dc3c7cdf4b0781fbfc18c270a3a5 P bfbbe1513aad4179abc52064d9250a9a: No bootstrap required, opened a new log
I20260812 02:34:53.600687 31476 ts_tablet_manager.cc:1397] T ca63dc3c7cdf4b0781fbfc18c270a3a5 P 3909f7c94d654754a9d589bb0ae09e8a: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260812 02:34:53.600700 31475 ts_tablet_manager.cc:1397] T ca63dc3c7cdf4b0781fbfc18c270a3a5 P bfbbe1513aad4179abc52064d9250a9a: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260812 02:34:53.604103 31476 raft_consensus.cc:348] T ca63dc3c7cdf4b0781fbfc18c270a3a5 P 3909f7c94d654754a9d589bb0ae09e8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3909f7c94d654754a9d589bb0ae09e8a" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 46173 } } peers { permanent_uuid: "bfbbe1513aad4179abc52064d9250a9a" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 32843 } } peers { permanent_uuid: "649a01114e0a46628578bff8843a050a" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 45955 } }
I20260812 02:34:53.604205 31475 raft_consensus.cc:348] T ca63dc3c7cdf4b0781fbfc18c270a3a5 P bfbbe1513aad4179abc52064d9250a9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3909f7c94d654754a9d589bb0ae09e8a" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 46173 } } peers { permanent_uuid: "bfbbe1513aad4179abc52064d9250a9a" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 32843 } } peers { permanent_uuid: "649a01114e0a46628578bff8843a050a" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 45955 } }
I20260812 02:34:53.604472 31476 raft_consensus.cc:374] T ca63dc3c7cdf4b0781fbfc18c270a3a5 P 3909f7c94d654754a9d589bb0ae09e8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:34:53.604529 31476 raft_consensus.cc:729] T ca63dc3c7cdf4b0781fbfc18c270a3a5 P 3909f7c94d654754a9d589bb0ae09e8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3909f7c94d654754a9d589bb0ae09e8a, State: Initialized, Role: FOLLOWER
I20260812 02:34:53.604526 31475 raft_consensus.cc:374] T ca63dc3c7cdf4b0781fbfc18c270a3a5 P bfbbe1513aad4179abc52064d9250a9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:34:53.604590 31475 raft_consensus.cc:729] T ca63dc3c7cdf4b0781fbfc18c270a3a5 P bfbbe1513aad4179abc52064d9250a9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bfbbe1513aad4179abc52064d9250a9a, State: Initialized, Role: FOLLOWER
I20260812 02:34:53.605113 31475 consensus_queue.cc:260] T ca63dc3c7cdf4b0781fbfc18c270a3a5 P bfbbe1513aad4179abc52064d9250a9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3909f7c94d654754a9d589bb0ae09e8a" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 46173 } } peers { permanent_uuid: "bfbbe1513aad4179abc52064d9250a9a" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 32843 } } peers { permanent_uuid: "649a01114e0a46628578bff8843a050a" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 45955 } }
I20260812 02:34:53.605110 31476 consensus_queue.cc:260] T ca63dc3c7cdf4b0781fbfc18c270a3a5 P 3909f7c94d654754a9d589bb0ae09e8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3909f7c94d654754a9d589bb0ae09e8a" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 46173 } } peers { permanent_uuid: "bfbbe1513aad4179abc52064d9250a9a" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 32843 } } peers { permanent_uuid: "649a01114e0a46628578bff8843a050a" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 45955 } }
I20260812 02:34:53.605865 31310 heartbeater.cc:503] Master 127.14.103.62:43307 was elected leader, sending a full tablet report...
I20260812 02:34:53.605865 31458 heartbeater.cc:503] Master 127.14.103.62:43307 was elected leader, sending a full tablet report...
I20260812 02:34:53.606299 31476 ts_tablet_manager.cc:1428] T ca63dc3c7cdf4b0781fbfc18c270a3a5 P 3909f7c94d654754a9d589bb0ae09e8a: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260812 02:34:53.606314 31475 ts_tablet_manager.cc:1428] T ca63dc3c7cdf4b0781fbfc18c270a3a5 P bfbbe1513aad4179abc52064d9250a9a: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260812 02:34:53.617619 31159 raft_consensus.cc:482] T ca63dc3c7cdf4b0781fbfc18c270a3a5 P 649a01114e0a46628578bff8843a050a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 02:34:53.617852 31159 raft_consensus.cc:504] T ca63dc3c7cdf4b0781fbfc18c270a3a5 P 649a01114e0a46628578bff8843a050a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3909f7c94d654754a9d589bb0ae09e8a" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 46173 } } peers { permanent_uuid: "bfbbe1513aad4179abc52064d9250a9a" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 32843 } } peers { permanent_uuid: "649a01114e0a46628578bff8843a050a" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 45955 } }
I20260812 02:34:53.619141 31159 leader_election.cc:288] T ca63dc3c7cdf4b0781fbfc18c270a3a5 P 649a01114e0a46628578bff8843a050a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3909f7c94d654754a9d589bb0ae09e8a (127.14.103.2:46173), bfbbe1513aad4179abc52064d9250a9a (127.14.103.3:32843)
I20260812 02:34:53.623219 31470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:53.619533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.1:58929 (local address 127.14.103.2:46173)
0812 02:34:53.619675 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0812 02:34:53.619679 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:34:53.619838 (+ 159us) server_negotiation.cc:407] Connection header received
0812 02:34:53.620041 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:34:53.620043 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:34:53.620091 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:34:53.620174 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:34:53.620460 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:53.620980 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:53.622149 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:53.622562 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:53.622702 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:53.622759 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0812 02:34:53.623008 (+ 249us) server_negotiation.cc:299] Negotiation successful
0812 02:34:53.623086 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260812 02:34:53.623514 31482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:53.619351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.2:46173 (local address 127.14.103.1:58929)
0812 02:34:53.619737 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0812 02:34:53.619749 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 02:34:53.619906 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:34:53.620195 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:34:53.620199 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:34:53.620216 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:34:53.620353 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:34:53.620359 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:34:53.621094 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:34:53.621099 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:34:53.622020 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:34:53.622030 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:34:53.622702 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:34:53.622706 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:34:53.622819 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:53.622868 (+ 49us) client_negotiation.cc:769] Sending connection context
0812 02:34:53.622956 (+ 88us) client_negotiation.cc:240] Negotiation successful
0812 02:34:53.623073 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":124,"threads_started":1}
I20260812 02:34:53.623646 31472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:53.619586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.1:44499 (local address 127.14.103.3:32843)
0812 02:34:53.619723 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0812 02:34:53.619729 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 02:34:53.619953 (+ 224us) server_negotiation.cc:407] Connection header received
0812 02:34:53.620089 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:34:53.620092 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:34:53.620323 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:34:53.620448 (+ 125us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:34:53.620728 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:53.621256 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:53.622471 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:53.622901 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:53.623076 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:53.623133 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0812 02:34:53.623416 (+ 283us) server_negotiation.cc:299] Negotiation successful
0812 02:34:53.623498 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260812 02:34:53.624065 31483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:53.619452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.3:32843 (local address 127.14.103.1:44499)
0812 02:34:53.619858 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0812 02:34:53.619871 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 02:34:53.619996 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:34:53.620462 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:34:53.620465 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:34:53.620483 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:34:53.620618 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:34:53.620624 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:34:53.621533 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:34:53.621537 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:34:53.622333 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:34:53.622342 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:34:53.623094 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:34:53.623098 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:34:53.623233 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:53.623254 (+ 21us) client_negotiation.cc:769] Sending connection context
0812 02:34:53.623803 (+ 549us) client_negotiation.cc:240] Negotiation successful
0812 02:34:53.623917 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":87,"threads_started":1}
I20260812 02:34:53.624229 31255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca63dc3c7cdf4b0781fbfc18c270a3a5" candidate_uuid: "649a01114e0a46628578bff8843a050a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3909f7c94d654754a9d589bb0ae09e8a" is_pre_election: true
I20260812 02:34:53.624642 31255 raft_consensus.cc:2454] T ca63dc3c7cdf4b0781fbfc18c270a3a5 P 3909f7c94d654754a9d589bb0ae09e8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 649a01114e0a46628578bff8843a050a in term 0.
I20260812 02:34:53.624716 31406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca63dc3c7cdf4b0781fbfc18c270a3a5" candidate_uuid: "649a01114e0a46628578bff8843a050a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bfbbe1513aad4179abc52064d9250a9a" is_pre_election: true
I20260812 02:34:53.625005 31406 raft_consensus.cc:2454] T ca63dc3c7cdf4b0781fbfc18c270a3a5 P bfbbe1513aad4179abc52064d9250a9a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 649a01114e0a46628578bff8843a050a in term 0.
I20260812 02:34:53.625247 31021 leader_election.cc:302] T ca63dc3c7cdf4b0781fbfc18c270a3a5 P 649a01114e0a46628578bff8843a050a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3909f7c94d654754a9d589bb0ae09e8a, 649a01114e0a46628578bff8843a050a; no voters:
I20260812 02:34:53.625588 31159 raft_consensus.cc:2781] T ca63dc3c7cdf4b0781fbfc18c270a3a5 P 649a01114e0a46628578bff8843a050a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 02:34:53.625675 31159 raft_consensus.cc:482] T ca63dc3c7cdf4b0781fbfc18c270a3a5 P 649a01114e0a46628578bff8843a050a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 02:34:53.625706 31159 raft_consensus.cc:3037] T ca63dc3c7cdf4b0781fbfc18c270a3a5 P 649a01114e0a46628578bff8843a050a [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:34:53.626925 31159 raft_consensus.cc:504] T ca63dc3c7cdf4b0781fbfc18c270a3a5 P 649a01114e0a46628578bff8843a050a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3909f7c94d654754a9d589bb0ae09e8a" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 46173 } } peers { permanent_uuid: "bfbbe1513aad4179abc52064d9250a9a" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 32843 } } peers { permanent_uuid: "649a01114e0a46628578bff8843a050a" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 45955 } }
I20260812 02:34:53.627388 31159 leader_election.cc:288] T ca63dc3c7cdf4b0781fbfc18c270a3a5 P 649a01114e0a46628578bff8843a050a [CANDIDATE]: Term 1 election: Requested vote from peers 3909f7c94d654754a9d589bb0ae09e8a (127.14.103.2:46173), bfbbe1513aad4179abc52064d9250a9a (127.14.103.3:32843)
I20260812 02:34:53.628024 31255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca63dc3c7cdf4b0781fbfc18c270a3a5" candidate_uuid: "649a01114e0a46628578bff8843a050a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3909f7c94d654754a9d589bb0ae09e8a"
I20260812 02:34:53.628196 31255 raft_consensus.cc:3037] T ca63dc3c7cdf4b0781fbfc18c270a3a5 P 3909f7c94d654754a9d589bb0ae09e8a [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:34:53.628228 31406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca63dc3c7cdf4b0781fbfc18c270a3a5" candidate_uuid: "649a01114e0a46628578bff8843a050a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bfbbe1513aad4179abc52064d9250a9a"
I20260812 02:34:53.628376 31406 raft_consensus.cc:3037] T ca63dc3c7cdf4b0781fbfc18c270a3a5 P bfbbe1513aad4179abc52064d9250a9a [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:34:53.629719 31406 raft_consensus.cc:2454] T ca63dc3c7cdf4b0781fbfc18c270a3a5 P bfbbe1513aad4179abc52064d9250a9a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 649a01114e0a46628578bff8843a050a in term 1.
I20260812 02:34:53.629719 31255 raft_consensus.cc:2454] T ca63dc3c7cdf4b0781fbfc18c270a3a5 P 3909f7c94d654754a9d589bb0ae09e8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 649a01114e0a46628578bff8843a050a in term 1.
I20260812 02:34:53.630103 31020 leader_election.cc:302] T ca63dc3c7cdf4b0781fbfc18c270a3a5 P 649a01114e0a46628578bff8843a050a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 3909f7c94d654754a9d589bb0ae09e8a, 649a01114e0a46628578bff8843a050a, bfbbe1513aad4179abc52064d9250a9a; no voters:
I20260812 02:34:53.630386 31159 raft_consensus.cc:2781] T ca63dc3c7cdf4b0781fbfc18c270a3a5 P 649a01114e0a46628578bff8843a050a [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:34:53.630607 31159 raft_consensus.cc:686] T ca63dc3c7cdf4b0781fbfc18c270a3a5 P 649a01114e0a46628578bff8843a050a [term 1 LEADER]: Becoming Leader. State: Replica: 649a01114e0a46628578bff8843a050a, State: Running, Role: LEADER
I20260812 02:34:53.630993 31159 consensus_queue.cc:237] T ca63dc3c7cdf4b0781fbfc18c270a3a5 P 649a01114e0a46628578bff8843a050a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3909f7c94d654754a9d589bb0ae09e8a" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 46173 } } peers { permanent_uuid: "bfbbe1513aad4179abc52064d9250a9a" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 32843 } } peers { permanent_uuid: "649a01114e0a46628578bff8843a050a" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 45955 } }
I20260812 02:34:53.634357 30930 catalog_manager.cc:5495] T ca63dc3c7cdf4b0781fbfc18c270a3a5 P 649a01114e0a46628578bff8843a050a reported cstate change: term changed from 0 to 1, leader changed from <none> to 649a01114e0a46628578bff8843a050a (127.14.103.1). New cstate: current_term: 1 leader_uuid: "649a01114e0a46628578bff8843a050a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3909f7c94d654754a9d589bb0ae09e8a" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 46173 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bfbbe1513aad4179abc52064d9250a9a" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 32843 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "649a01114e0a46628578bff8843a050a" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 45955 } health_report { overall_health: HEALTHY } } }
I20260812 02:34:53.645781 31486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:53.641649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54416 (local address 127.14.103.1:45955)
0812 02:34:53.641928 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0812 02:34:53.641931 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 02:34:53.641945 (+ 14us) server_negotiation.cc:407] Connection header received
0812 02:34:53.642020 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:34:53.642023 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:34:53.642077 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:34:53.642181 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 02:34:53.642441 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:53.643062 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:53.644073 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:53.644378 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:53.644482 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:53.644752 (+ 270us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 02:34:53.645308 (+ 556us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 02:34:53.645404 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0812 02:34:53.645533 (+ 129us) server_negotiation.cc:299] Negotiation successful
0812 02:34:53.645610 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":205,"thread_start_us":117,"threads_started":1}
I20260812 02:34:53.650586 31406 raft_consensus.cc:1261] T ca63dc3c7cdf4b0781fbfc18c270a3a5 P bfbbe1513aad4179abc52064d9250a9a [term 1 FOLLOWER]: Refusing update from remote peer 649a01114e0a46628578bff8843a050a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:34:53.650586 31255 raft_consensus.cc:1261] T ca63dc3c7cdf4b0781fbfc18c270a3a5 P 3909f7c94d654754a9d589bb0ae09e8a [term 1 FOLLOWER]: Refusing update from remote peer 649a01114e0a46628578bff8843a050a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:34:53.651468 31159 consensus_queue.cc:1035] T ca63dc3c7cdf4b0781fbfc18c270a3a5 P 649a01114e0a46628578bff8843a050a [LEADER]: Connected to new peer: Peer: permanent_uuid: "bfbbe1513aad4179abc52064d9250a9a" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 32843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:34:53.651707 31484 consensus_queue.cc:1035] T ca63dc3c7cdf4b0781fbfc18c270a3a5 P 649a01114e0a46628578bff8843a050a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3909f7c94d654754a9d589bb0ae09e8a" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 46173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:34:53.660504 31489 mvcc.cc:205] Tried to move back new op lower bound from 7317512575584845824 to 7317512575515422720. Current Snapshot: MvccSnapshot[applied={T|T < 7317512575584845824}]
I20260812 02:34:53.660887 31488 mvcc.cc:205] Tried to move back new op lower bound from 7317512575584845824 to 7317512575515422720. Current Snapshot: MvccSnapshot[applied={T|T < 7317512575584845824}]
I20260812 02:34:53.661430 31487 mvcc.cc:205] Tried to move back new op lower bound from 7317512575584845824 to 7317512575515422720. Current Snapshot: MvccSnapshot[applied={T|T < 7317512575584845824}]
I20260812 02:34:53.670133 30930 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:33000:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 02:34:53.670344 30930 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:33000:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 02:34:53.673233 30930 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d92eab9524324b82b329fd272df49540: Sending DeleteTablet for 3 replicas of tablet ca63dc3c7cdf4b0781fbfc18c270a3a5
I20260812 02:34:53.674217 31235 tablet_service.cc:1514] Processing DeleteTablet for tablet ca63dc3c7cdf4b0781fbfc18c270a3a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:34:53 UTC) from {username='kudu', principal='kudu/127.14.103.62@KRBTEST.COM'} at 127.0.0.1:39614
I20260812 02:34:53.674194 31384 tablet_service.cc:1514] Processing DeleteTablet for tablet ca63dc3c7cdf4b0781fbfc18c270a3a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:34:53 UTC) from {username='kudu', principal='kudu/127.14.103.62@KRBTEST.COM'} at 127.0.0.1:58272
I20260812 02:34:53.674386 31073 tablet_service.cc:1514] Processing DeleteTablet for tablet ca63dc3c7cdf4b0781fbfc18c270a3a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:34:53 UTC) from {username='kudu', principal='kudu/127.14.103.62@KRBTEST.COM'} at 127.0.0.1:54408
I20260812 02:34:53.674525 14748 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbep4M4/build/debug/bin/kudu with pid 30979
I20260812 02:34:53.679822 31503 tablet_replica.cc:330] T ca63dc3c7cdf4b0781fbfc18c270a3a5 P bfbbe1513aad4179abc52064d9250a9a: stopping tablet replica
I20260812 02:34:53.680250 31503 raft_consensus.cc:2229] T ca63dc3c7cdf4b0781fbfc18c270a3a5 P bfbbe1513aad4179abc52064d9250a9a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 02:34:53.680637 31503 raft_consensus.cc:2258] T ca63dc3c7cdf4b0781fbfc18c270a3a5 P bfbbe1513aad4179abc52064d9250a9a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:34:53.681118 31502 tablet_replica.cc:330] T ca63dc3c7cdf4b0781fbfc18c270a3a5 P 3909f7c94d654754a9d589bb0ae09e8a: stopping tablet replica
I20260812 02:34:53.681485 31502 raft_consensus.cc:2229] T ca63dc3c7cdf4b0781fbfc18c270a3a5 P 3909f7c94d654754a9d589bb0ae09e8a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 02:34:53.681802 31502 raft_consensus.cc:2258] T ca63dc3c7cdf4b0781fbfc18c270a3a5 P 3909f7c94d654754a9d589bb0ae09e8a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:34:53.682530 31503 ts_tablet_manager.cc:1905] T ca63dc3c7cdf4b0781fbfc18c270a3a5 P bfbbe1513aad4179abc52064d9250a9a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:34:53.683185 31502 ts_tablet_manager.cc:1905] T ca63dc3c7cdf4b0781fbfc18c270a3a5 P 3909f7c94d654754a9d589bb0ae09e8a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:34:53.685457 31502 ts_tablet_manager.cc:1918] T ca63dc3c7cdf4b0781fbfc18c270a3a5 P 3909f7c94d654754a9d589bb0ae09e8a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 02:34:53.685606 31502 log.cc:1196] T ca63dc3c7cdf4b0781fbfc18c270a3a5 P 3909f7c94d654754a9d589bb0ae09e8a: Deleting WAL directory at /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/minicluster-data/ts-1/wal/wals/ca63dc3c7cdf4b0781fbfc18c270a3a5
I20260812 02:34:53.685986 31502 ts_tablet_manager.cc:1939] T ca63dc3c7cdf4b0781fbfc18c270a3a5 P 3909f7c94d654754a9d589bb0ae09e8a: Deleting consensus metadata
I20260812 02:34:53.686427 31503 ts_tablet_manager.cc:1918] T ca63dc3c7cdf4b0781fbfc18c270a3a5 P bfbbe1513aad4179abc52064d9250a9a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 02:34:53.686558 31503 log.cc:1196] T ca63dc3c7cdf4b0781fbfc18c270a3a5 P bfbbe1513aad4179abc52064d9250a9a: Deleting WAL directory at /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/minicluster-data/ts-2/wal/wals/ca63dc3c7cdf4b0781fbfc18c270a3a5
I20260812 02:34:53.686920 31503 ts_tablet_manager.cc:1939] T ca63dc3c7cdf4b0781fbfc18c270a3a5 P bfbbe1513aad4179abc52064d9250a9a: Deleting consensus metadata
W20260812 02:34:53.687288 30917 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.103.1:45955 (error 108)
I20260812 02:34:53.688205 30916 catalog_manager.cc:4841] TS bfbbe1513aad4179abc52064d9250a9a (127.14.103.3:32843): tablet ca63dc3c7cdf4b0781fbfc18c270a3a5 (table test-table [id=a9e33cc803e44b969a8585e4f784818a]) successfully deleted
I20260812 02:34:53.688483 31468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:53.688114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.1:45955 (local address 127.0.0.1:54424)
0812 02:34:53.688323 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0812 02:34:53.688392 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.1:45955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128}
I20260812 02:34:53.688231 30917 catalog_manager.cc:4841] TS 3909f7c94d654754a9d589bb0ae09e8a (127.14.103.2:46173): tablet ca63dc3c7cdf4b0781fbfc18c270a3a5 (table test-table [id=a9e33cc803e44b969a8585e4f784818a]) successfully deleted
I20260812 02:34:53.689039 14748 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbep4M4/build/debug/bin/kudu with pid 31157
W20260812 02:34:53.688730 30917 catalog_manager.cc:4584] TS 649a01114e0a46628578bff8843a050a (127.14.103.1:45955): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ca63dc3c7cdf4b0781fbfc18c270a3a5: Network error: Client connection negotiation failed: client connection to 127.14.103.1:45955: connect: Connection refused (error 111)
I20260812 02:34:53.698580 14748 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbep4M4/build/debug/bin/kudu with pid 31314
I20260812 02:34:53.708845 14748 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbep4M4/build/debug/bin/kudu with pid 30897
I20260812 02:34:53.720378 14748 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbep4M4/build/debug/bin/kudu
/tmp/dist-test-taskbep4M4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.103.62:43307
--webserver_interface=127.14.103.62
--webserver_port=36911
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.62.keytab
--principal=kudu/127.14.103.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.103.20:36423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.103.62:43307
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:34:53.844327 31506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:34:53.844640 31506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:34:53.844681 31506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:34:53.849104 31506 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:34:53.849205 31506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:34:53.849226 31506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:34:53.849247 31506 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:34:53.849263 31506 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 02:34:53.855847 31506 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.103.20:36423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.103.62:43307
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.62.keytab
--principal=kudu/127.14.103.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.103.62:43307
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.103.62
--webserver_port=36911
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:34:53.857014 31506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:34:53.858035 31506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:34:53.867357 31514 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:34:53.867611 31506 server_base.cc:1034] running on GCE node
W20260812 02:34:53.867329 31516 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:34:53.867329 31511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:34:53.868321 31506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:34:53.869478 31506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:34:53.870656 31506 hybrid_clock.cc:648] HybridClock initialized: now 1786502093870623 us; error 48 us; skew 500 ppm
Aug 12 02:34:53 dist-test-slave-2w3w krb5kdc[30263](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502093, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:34:53.873939 31506 init.cc:377] Logged in from keytab as kudu/127.14.103.62@KRBTEST.COM (short username kudu)
I20260812 02:34:53.875134 31506 webserver.cc:460] Webserver started at http://127.14.103.62:36911/ using document root <none> and password file <none>
I20260812 02:34:53.875771 31506 fs_manager.cc:359] Metadata directory not provided
I20260812 02:34:53.875855 31506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:34:53.880466 31506 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20260812 02:34:53.883373 31524 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:34:53.884819 31506 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260812 02:34:53.885000 31506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/minicluster-data/master-0/data,/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/minicluster-data/master-0/wal
uuid: "d92eab9524324b82b329fd272df49540"
format_stamp: "Formatted at 2026-08-12 02:34:51 on dist-test-slave-2w3w"
server_key: "664a157ae4b40297525e3039b779c47b"
server_key_iv: "295b86a832b5ed934d9001353bd656fc"
server_key_version: "encryptionkey@0"
I20260812 02:34:53.885612 31506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:34:53.900884 31506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:34:53.904515 31506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:34:53.904745 31506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:34:53.912428 31506 rpc_server.cc:307] RPC server started. Bound to: 127.14.103.62:43307
I20260812 02:34:53.912750 31583 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.103.62:43307 every 8 connection(s)
I20260812 02:34:53.913641 31506 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/minicluster-data/master-0/data/info.pb
I20260812 02:34:53.918289 31584 sys_catalog.cc:263] Verifying existing consensus state
I20260812 02:34:53.918898 14748 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbep4M4/build/debug/bin/kudu as pid 31506
I20260812 02:34:53.919451 14748 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbep4M4/build/debug/bin/kudu
/tmp/dist-test-taskbep4M4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.103.1:45955
--local_ip_for_outbound_sockets=127.14.103.1
--tserver_master_addrs=127.14.103.62:43307
--webserver_port=44913
--webserver_interface=127.14.103.1
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.1.keytab
--principal=kudu/127.14.103.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.103.20:36423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 02:34:53.920758 31584 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d92eab9524324b82b329fd272df49540: Bootstrap starting.
I20260812 02:34:53.941195 31584 log.cc:824] T 00000000000000000000000000000000 P d92eab9524324b82b329fd272df49540: Log is configured to *not* fsync() on all Append() calls
I20260812 02:34:53.953647 31584 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d92eab9524324b82b329fd272df49540: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 02:34:53.954319 31584 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d92eab9524324b82b329fd272df49540: Bootstrap complete.
I20260812 02:34:53.959209 31584 raft_consensus.cc:348] T 00000000000000000000000000000000 P d92eab9524324b82b329fd272df49540 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d92eab9524324b82b329fd272df49540" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 43307 } }
I20260812 02:34:53.959681 31584 raft_consensus.cc:729] T 00000000000000000000000000000000 P d92eab9524324b82b329fd272df49540 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: d92eab9524324b82b329fd272df49540, State: Initialized, Role: FOLLOWER
I20260812 02:34:53.960412 31584 consensus_queue.cc:260] T 00000000000000000000000000000000 P d92eab9524324b82b329fd272df49540 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d92eab9524324b82b329fd272df49540" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 43307 } }
I20260812 02:34:53.960608 31584 raft_consensus.cc:388] T 00000000000000000000000000000000 P d92eab9524324b82b329fd272df49540 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 02:34:53.960692 31584 raft_consensus.cc:482] T 00000000000000000000000000000000 P d92eab9524324b82b329fd272df49540 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 02:34:53.960829 31584 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d92eab9524324b82b329fd272df49540 [term 2 FOLLOWER]: Advancing to term 3
I20260812 02:34:53.963728 31584 raft_consensus.cc:504] T 00000000000000000000000000000000 P d92eab9524324b82b329fd272df49540 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d92eab9524324b82b329fd272df49540" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 43307 } }
I20260812 02:34:53.964170 31584 leader_election.cc:302] T 00000000000000000000000000000000 P d92eab9524324b82b329fd272df49540 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d92eab9524324b82b329fd272df49540; no voters:
I20260812 02:34:53.964589 31584 leader_election.cc:288] T 00000000000000000000000000000000 P d92eab9524324b82b329fd272df49540 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260812 02:34:53.964725 31588 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d92eab9524324b82b329fd272df49540 [term 3 FOLLOWER]: Leader election won for term 3
I20260812 02:34:53.964972 31588 raft_consensus.cc:686] T 00000000000000000000000000000000 P d92eab9524324b82b329fd272df49540 [term 3 LEADER]: Becoming Leader. State: Replica: d92eab9524324b82b329fd272df49540, State: Running, Role: LEADER
I20260812 02:34:53.965426 31588 consensus_queue.cc:237] T 00000000000000000000000000000000 P d92eab9524324b82b329fd272df49540 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d92eab9524324b82b329fd272df49540" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 43307 } }
I20260812 02:34:53.966188 31584 sys_catalog.cc:564] T 00000000000000000000000000000000 P d92eab9524324b82b329fd272df49540 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:34:53.967381 31589 sys_catalog.cc:455] T 00000000000000000000000000000000 P d92eab9524324b82b329fd272df49540 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "d92eab9524324b82b329fd272df49540" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d92eab9524324b82b329fd272df49540" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 43307 } } }
I20260812 02:34:53.967567 31589 sys_catalog.cc:458] T 00000000000000000000000000000000 P d92eab9524324b82b329fd272df49540 [sys.catalog]: This master's current role is: LEADER
I20260812 02:34:53.967911 31590 sys_catalog.cc:455] T 00000000000000000000000000000000 P d92eab9524324b82b329fd272df49540 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d92eab9524324b82b329fd272df49540. Latest consensus state: current_term: 3 leader_uuid: "d92eab9524324b82b329fd272df49540" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d92eab9524324b82b329fd272df49540" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 43307 } } }
I20260812 02:34:53.968029 31590 sys_catalog.cc:458] T 00000000000000000000000000000000 P d92eab9524324b82b329fd272df49540 [sys.catalog]: This master's current role is: LEADER
I20260812 02:34:53.968025 31600 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 02:34:53.974552 31600 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 02:34:53.975513 31600 catalog_manager.cc:1259] Loaded cluster ID: fdb7ae612c83410bbce5c0d124a4977b
I20260812 02:34:53.975587 31600 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260812 02:34:53.976603 31600 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f9b125469ad google::LogMessage::Fail() at ??:0
@ 0x7f9b1254988a google::LogMessage::SendToLog() at ??:0
@ 0x7f9b125464de google::LogMessage::Flush() at ??:0
@ 0x7f9b12547a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f9b1ce85fdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f9b1ce86d2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f9b1ce83579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f9b1cec7640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f9b147983d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f9b14798c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f9b1479a3f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f9b14789512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f9b15dcd6db start_thread at ??:0
@ 0x7f9b1104461f clone at ??:0
W20260812 02:34:54.063727 31586 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:34:54.063997 31586 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:34:54.064042 31586 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:34:54.068486 31586 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:34:54.068589 31586 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:34:54.068702 31586 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.103.1
I20260812 02:34:54.074359 31586 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.103.20:36423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.1.keytab
--principal=kudu/127.14.103.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.103.1:45955
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.103.1
--webserver_port=44913
--tserver_master_addrs=127.14.103.62:43307
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31586
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.103.1
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:34:54.075604 31586 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:34:54.076669 31586 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:34:54.087265 31631 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:34:54.087352 31632 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:34:54.087582 31635 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:34:54.087566 31586 server_base.cc:1034] running on GCE node
I20260812 02:34:54.088209 31586 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:34:54.089112 31586 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:34:54.090343 31586 hybrid_clock.cc:648] HybridClock initialized: now 1786502094090318 us; error 43 us; skew 500 ppm
Aug 12 02:34:54 dist-test-slave-2w3w krb5kdc[30263](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502094, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:34:54.094607 31586 init.cc:377] Logged in from keytab as kudu/127.14.103.1@KRBTEST.COM (short username kudu)
I20260812 02:34:54.095878 31586 webserver.cc:460] Webserver started at http://127.14.103.1:44913/ using document root <none> and password file <none>
I20260812 02:34:54.096555 31586 fs_manager.cc:359] Metadata directory not provided
I20260812 02:34:54.096661 31586 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:34:54.101801 31586 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260812 02:34:54.104485 31642 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:34:54.106220 31586 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260812 02:34:54.106383 31586 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/minicluster-data/ts-0/wal
uuid: "649a01114e0a46628578bff8843a050a"
format_stamp: "Formatted at 2026-08-12 02:34:51 on dist-test-slave-2w3w"
server_key: "635812a0cc12b707e20ed3264aacc3ea"
server_key_iv: "37d30bf5c49236f63ef0991a57da38d0"
server_key_version: "encryptionkey@0"
I20260812 02:34:54.107019 31586 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:34:54.123373 31586 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:34:54.127506 31586 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:34:54.127767 31586 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:34:54.128515 31586 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:34:54.129956 31649 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260812 02:34:54.136097 31586 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260812 02:34:54.136233 31586 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.007s user 0.001s sys 0.000s
I20260812 02:34:54.136317 31586 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260812 02:34:54.138234 31586 ts_tablet_manager.cc:610] Registered 1 tablets
I20260812 02:34:54.138329 31586 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260812 02:34:54.138659 31649 tablet_bootstrap.cc:492] T ca63dc3c7cdf4b0781fbfc18c270a3a5 P 649a01114e0a46628578bff8843a050a: Bootstrap starting.
I20260812 02:34:54.150861 31586 rpc_server.cc:307] RPC server started. Bound to: 127.14.103.1:45955
I20260812 02:34:54.150888 31772 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.103.1:45955 every 8 connection(s)
I20260812 02:34:54.152030 31586 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/minicluster-data/ts-0/data/info.pb
I20260812 02:34:54.159066 14748 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbep4M4/build/debug/bin/kudu as pid 31586
I20260812 02:34:54.159536 14748 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbep4M4/build/debug/bin/kudu
/tmp/dist-test-taskbep4M4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.103.2:46173
--local_ip_for_outbound_sockets=127.14.103.2
--tserver_master_addrs=127.14.103.62:43307
--webserver_port=34039
--webserver_interface=127.14.103.2
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.2.keytab
--principal=kudu/127.14.103.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.103.20:36423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 02:34:54.162756 31649 log.cc:824] T ca63dc3c7cdf4b0781fbfc18c270a3a5 P 649a01114e0a46628578bff8843a050a: Log is configured to *not* fsync() on all Append() calls
I20260812 02:34:54.173715 31649 tablet_bootstrap.cc:492] T ca63dc3c7cdf4b0781fbfc18c270a3a5 P 649a01114e0a46628578bff8843a050a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 02:34:54.174198 31649 tablet_bootstrap.cc:492] T ca63dc3c7cdf4b0781fbfc18c270a3a5 P 649a01114e0a46628578bff8843a050a: Bootstrap complete.
I20260812 02:34:54.175629 31649 ts_tablet_manager.cc:1397] T ca63dc3c7cdf4b0781fbfc18c270a3a5 P 649a01114e0a46628578bff8843a050a: Time spent bootstrapping tablet: real 0.037s user 0.024s sys 0.010s
I20260812 02:34:54.178244 31649 raft_consensus.cc:348] T ca63dc3c7cdf4b0781fbfc18c270a3a5 P 649a01114e0a46628578bff8843a050a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3909f7c94d654754a9d589bb0ae09e8a" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 46173 } } peers { permanent_uuid: "bfbbe1513aad4179abc52064d9250a9a" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 32843 } } peers { permanent_uuid: "649a01114e0a46628578bff8843a050a" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 45955 } }
I20260812 02:34:54.178736 31649 raft_consensus.cc:729] T ca63dc3c7cdf4b0781fbfc18c270a3a5 P 649a01114e0a46628578bff8843a050a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 649a01114e0a46628578bff8843a050a, State: Initialized, Role: FOLLOWER
I20260812 02:34:54.179343 31649 consensus_queue.cc:260] T ca63dc3c7cdf4b0781fbfc18c270a3a5 P 649a01114e0a46628578bff8843a050a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3909f7c94d654754a9d589bb0ae09e8a" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 46173 } } peers { permanent_uuid: "bfbbe1513aad4179abc52064d9250a9a" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 32843 } } peers { permanent_uuid: "649a01114e0a46628578bff8843a050a" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 45955 } }
I20260812 02:34:54.180390 31649 ts_tablet_manager.cc:1428] T ca63dc3c7cdf4b0781fbfc18c270a3a5 P 649a01114e0a46628578bff8843a050a: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
W20260812 02:34:54.250497 31778 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:34:54.154519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:43307 (local address 127.14.103.1:52067)
0812 02:34:54.155100 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0812 02:34:54.155162 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0812 02:34:54.156022 (+ 860us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:34:54.249625 (+ 93603us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.62:43307: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":196,"threads_started":1}
W20260812 02:34:54.251302 31774 heartbeater.cc:644] Failed to heartbeat to 127.14.103.62:43307 (0 consecutive failures): Network error: Failed to ping master at 127.14.103.62:43307: Client connection negotiation failed: client connection to 127.14.103.62:43307: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260812 02:34:54.252220 31778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:54.251862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:43307 (local address 127.14.103.1:59809)
0812 02:34:54.252039 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0812 02:34:54.252137 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.62:43307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260812 02:34:54.253087 31778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:54.252821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:43307 (local address 127.14.103.1:49527)
0812 02:34:54.252952 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0812 02:34:54.253011 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.62:43307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260812 02:34:54.253328 31774 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260812 02:34:54.305847 31779 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:34:54.306162 31779 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:34:54.306210 31779 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:34:54.313231 31779 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:34:54.313334 31779 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:34:54.313409 31779 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.103.2
I20260812 02:34:54.320079 31779 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.103.20:36423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.2.keytab
--principal=kudu/127.14.103.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.103.2:46173
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.103.2
--webserver_port=34039
--tserver_master_addrs=127.14.103.62:43307
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31779
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.103.2
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:34:54.321321 31779 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:34:54.322373 31779 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:34:54.333756 31788 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:34:54.333698 31790 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:34:54.333698 31787 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:34:54.334167 31779 server_base.cc:1034] running on GCE node
I20260812 02:34:54.334600 31779 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:34:54.335435 31779 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:34:54.336632 31779 hybrid_clock.cc:648] HybridClock initialized: now 1786502094336609 us; error 36 us; skew 500 ppm
Aug 12 02:34:54 dist-test-slave-2w3w krb5kdc[30263](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502094, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:34:54.340047 31779 init.cc:377] Logged in from keytab as kudu/127.14.103.2@KRBTEST.COM (short username kudu)
I20260812 02:34:54.341112 31779 webserver.cc:460] Webserver started at http://127.14.103.2:34039/ using document root <none> and password file <none>
I20260812 02:34:54.341722 31779 fs_manager.cc:359] Metadata directory not provided
I20260812 02:34:54.341789 31779 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:34:54.346098 31779 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260812 02:34:54.348577 31798 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:34:54.350086 31779 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260812 02:34:54.350253 31779 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/minicluster-data/ts-1/wal
uuid: "3909f7c94d654754a9d589bb0ae09e8a"
format_stamp: "Formatted at 2026-08-12 02:34:52 on dist-test-slave-2w3w"
server_key: "2f5c8300d53314fffe9ae0ed32f4ad3e"
server_key_iv: "d2bd2e696ab65ca38afef510784b98ef"
server_key_version: "encryptionkey@0"
I20260812 02:34:54.351094 31779 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:34:54.373950 31779 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:34:54.377640 31779 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:34:54.377995 31779 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:34:54.378716 31779 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:34:54.379796 31779 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:34:54.379871 31779 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:34:54.379943 31779 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:34:54.379983 31779 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:34:54.390496 31779 rpc_server.cc:307] RPC server started. Bound to: 127.14.103.2:46173
I20260812 02:34:54.390533 31928 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.103.2:46173 every 8 connection(s)
I20260812 02:34:54.391651 31779 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/minicluster-data/ts-1/data/info.pb
I20260812 02:34:54.395759 31932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:54.393968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:43307 (local address 127.14.103.2:55129)
0812 02:34:54.394479 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0812 02:34:54.394946 (+ 467us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.62:43307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":115,"threads_started":1}
W20260812 02:34:54.396302 31929 heartbeater.cc:644] Failed to heartbeat to 127.14.103.62:43307 (0 consecutive failures): Network error: Failed to ping master at 127.14.103.62:43307: Client connection negotiation failed: client connection to 127.14.103.62:43307: connect: Connection refused (error 111)
I20260812 02:34:54.397015 31932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:54.396729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:43307 (local address 127.14.103.2:38865)
0812 02:34:54.396872 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0812 02:34:54.396956 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.62:43307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260812 02:34:54.397780 31932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:54.397583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:43307 (local address 127.14.103.2:41879)
0812 02:34:54.397680 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0812 02:34:54.397726 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.62:43307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260812 02:34:54.397998 31929 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260812 02:34:54.400175 14748 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbep4M4/build/debug/bin/kudu as pid 31779
I20260812 02:34:54.400615 14748 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbep4M4/build/debug/bin/kudu
/tmp/dist-test-taskbep4M4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.103.3:32843
--local_ip_for_outbound_sockets=127.14.103.3
--tserver_master_addrs=127.14.103.62:43307
--webserver_port=33687
--webserver_interface=127.14.103.3
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.3.keytab
--principal=kudu/127.14.103.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.103.20:36423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:34:54.526388 31933 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:34:54.526698 31933 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:34:54.526746 31933 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:34:54.531299 31933 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:34:54.531394 31933 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:34:54.531461 31933 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.103.3
I20260812 02:34:54.537169 31933 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.103.20:36423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.3.keytab
--principal=kudu/127.14.103.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.103.3:32843
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.103.3
--webserver_port=33687
--tserver_master_addrs=127.14.103.62:43307
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31933
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.103.3
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:34:54.538398 31933 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:34:54.539474 31933 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:34:54.550009 31939 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:34:54.550081 31933 server_base.cc:1034] running on GCE node
W20260812 02:34:54.550037 31941 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:34:54.550033 31938 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:34:54.550740 31933 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:34:54.551550 31933 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:34:54.552735 31933 hybrid_clock.cc:648] HybridClock initialized: now 1786502094552676 us; error 74 us; skew 500 ppm
Aug 12 02:34:54 dist-test-slave-2w3w krb5kdc[30263](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502094, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:34:54.556408 31933 init.cc:377] Logged in from keytab as kudu/127.14.103.3@KRBTEST.COM (short username kudu)
I20260812 02:34:54.557544 31933 webserver.cc:460] Webserver started at http://127.14.103.3:33687/ using document root <none> and password file <none>
I20260812 02:34:54.558368 31933 fs_manager.cc:359] Metadata directory not provided
I20260812 02:34:54.558475 31933 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:34:54.563426 31933 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260812 02:34:54.566097 31948 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:34:54.567914 31933 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 02:34:54.568084 31933 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/minicluster-data/ts-2/wal
uuid: "bfbbe1513aad4179abc52064d9250a9a"
format_stamp: "Formatted at 2026-08-12 02:34:52 on dist-test-slave-2w3w"
server_key: "5a89d8ac702231998b7f2134835d7d8c"
server_key_iv: "ab713679a6525a8ff3e27f67fb2b5a2b"
server_key_version: "encryptionkey@0"
I20260812 02:34:54.568684 31933 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:34:54.594293 31933 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:34:54.598768 31933 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:34:54.599368 31933 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:34:54.600267 31933 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:34:54.601230 31933 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:34:54.601282 31933 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:34:54.601325 31933 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:34:54.601387 31933 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:34:54.612058 31933 rpc_server.cc:307] RPC server started. Bound to: 127.14.103.3:32843
I20260812 02:34:54.612108 32081 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.103.3:32843 every 8 connection(s)
I20260812 02:34:54.613245 31933 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786501898713699-14748-0/minicluster-data/ts-2/data/info.pb
I20260812 02:34:54.617977 32085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:54.615891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:43307 (local address 127.14.103.3:41721)
0812 02:34:54.616685 (+ 794us) negotiation.cc:107] Waiting for socket to connect
0812 02:34:54.617175 (+ 490us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.62:43307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":549,"thread_start_us":134,"threads_started":1}
W20260812 02:34:54.618548 32083 heartbeater.cc:644] Failed to heartbeat to 127.14.103.62:43307 (0 consecutive failures): Network error: Failed to ping master at 127.14.103.62:43307: Client connection negotiation failed: client connection to 127.14.103.62:43307: connect: Connection refused (error 111)
I20260812 02:34:54.619499 32085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:54.619072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:43307 (local address 127.14.103.3:49205)
0812 02:34:54.619376 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0812 02:34:54.619434 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.62:43307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213}
I20260812 02:34:54.620478 32085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:54.620175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:43307 (local address 127.14.103.3:39123)
0812 02:34:54.620313 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0812 02:34:54.620374 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.62:43307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260812 02:34:54.620613 14748 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbep4M4/build/debug/bin/kudu as pid 31933
W20260812 02:34:54.620883 32083 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260812 02:34:55.254436 32087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:55.253958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:43307 (local address 127.14.103.1:51593)
0812 02:34:55.254235 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0812 02:34:55.254295 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.62:43307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":104,"threads_started":1}
I20260812 02:34:55.399432 32088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:55.398736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:43307 (local address 127.14.103.2:44999)
0812 02:34:55.399253 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0812 02:34:55.399316 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.62:43307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":236,"threads_started":1}
I20260812 02:34:55.622491 32089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:55.621952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:43307 (local address 127.14.103.3:46939)
0812 02:34:55.622294 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0812 02:34:55.622363 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.62:43307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":91,"threads_started":1}
I20260812 02:34:55.790462 32090 raft_consensus.cc:482] T ca63dc3c7cdf4b0781fbfc18c270a3a5 P 649a01114e0a46628578bff8843a050a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 02:34:55.790717 32090 raft_consensus.cc:504] T ca63dc3c7cdf4b0781fbfc18c270a3a5 P 649a01114e0a46628578bff8843a050a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3909f7c94d654754a9d589bb0ae09e8a" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 46173 } } peers { permanent_uuid: "bfbbe1513aad4179abc52064d9250a9a" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 32843 } } peers { permanent_uuid: "649a01114e0a46628578bff8843a050a" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 45955 } }
I20260812 02:34:55.791986 32090 leader_election.cc:288] T ca63dc3c7cdf4b0781fbfc18c270a3a5 P 649a01114e0a46628578bff8843a050a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3909f7c94d654754a9d589bb0ae09e8a (127.14.103.2:46173), bfbbe1513aad4179abc52064d9250a9a (127.14.103.3:32843)
Aug 12 02:34:55 dist-test-slave-2w3w krb5kdc[30263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502094, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.1@KRBTEST.COM for kudu/127.14.103.3@KRBTEST.COM
Aug 12 02:34:55 dist-test-slave-2w3w krb5kdc[30263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502094, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.1@KRBTEST.COM for kudu/127.14.103.2@KRBTEST.COM
I20260812 02:34:55.813308 32093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:55.792446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.1:33843 (local address 127.14.103.2:46173)
0812 02:34:55.792880 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0812 02:34:55.792909 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0812 02:34:55.792958 (+ 49us) server_negotiation.cc:407] Connection header received
0812 02:34:55.793790 (+ 832us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:34:55.793798 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:34:55.794300 (+ 502us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:34:55.794677 (+ 377us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:34:55.796173 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:55.797190 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:55.798551 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:55.799110 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:55.799297 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:55.807179 (+ 7882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:34:55.807185 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:34:55.807197 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:34:55.807231 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:34:55.809573 (+ 2342us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:34:55.810267 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:34:55.810273 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:34:55.810277 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:34:55.810576 (+ 299us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:34:55.810920 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:34:55.810923 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:34:55.810924 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:34:55.811439 (+ 515us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:34:55.811591 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0812 02:34:55.812478 (+ 887us) server_negotiation.cc:299] Negotiation successful
0812 02:34:55.813005 (+ 527us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":86,"threads_started":1}
I20260812 02:34:55.813295 32091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:55.792239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.2:46173 (local address 127.14.103.1:33843)
0812 02:34:55.792700 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0812 02:34:55.792717 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 02:34:55.792835 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:34:55.794904 (+ 2069us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:34:55.794912 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:34:55.795485 (+ 573us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:34:55.795982 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:34:55.795999 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:34:55.797379 (+ 1380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:34:55.797381 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:34:55.798411 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:34:55.798422 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:34:55.799242 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:34:55.799246 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:34:55.799529 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:55.804327 (+ 4798us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:34:55.804388 (+ 61us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:34:55.806840 (+ 2452us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:34:55.809713 (+ 2873us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:34:55.809716 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:34:55.809718 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:34:55.810069 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:34:55.810701 (+ 632us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:34:55.810704 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:34:55.810705 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:34:55.810787 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:34:55.811569 (+ 782us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:34:55.811579 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:34:55.811936 (+ 357us) client_negotiation.cc:769] Sending connection context
0812 02:34:55.812367 (+ 431us) client_negotiation.cc:240] Negotiation successful
0812 02:34:55.813157 (+ 790us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":373,"mutex_wait_us":567,"spinlock_wait_cycles":34304,"thread_start_us":92,"threads_started":1}
I20260812 02:34:55.813304 32092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:55.792348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.3:32843 (local address 127.14.103.1:60291)
0812 02:34:55.792896 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0812 02:34:55.792911 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 02:34:55.792999 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:34:55.795136 (+ 2137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:34:55.795139 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:34:55.795483 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:34:55.795982 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:34:55.795998 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:34:55.797368 (+ 1370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:34:55.797376 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:34:55.798413 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:34:55.798423 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:34:55.799240 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:34:55.799244 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:34:55.799530 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:55.804327 (+ 4797us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:34:55.804376 (+ 49us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:34:55.806505 (+ 2129us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:34:55.809607 (+ 3102us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:34:55.809617 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:34:55.809636 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:34:55.810041 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:34:55.810533 (+ 492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:34:55.810537 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:34:55.810539 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:34:55.810746 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:34:55.811751 (+ 1005us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:34:55.811754 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:34:55.811936 (+ 182us) client_negotiation.cc:769] Sending connection context
0812 02:34:55.812364 (+ 428us) client_negotiation.cc:240] Negotiation successful
0812 02:34:55.813078 (+ 714us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465,"mutex_wait_us":211,"spinlock_wait_cycles":109056,"thread_start_us":85,"threads_started":1}
I20260812 02:34:55.814420 32094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:55.792495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.1:60291 (local address 127.14.103.3:32843)
0812 02:34:55.793106 (+ 611us) server_negotiation.cc:206] Beginning negotiation
0812 02:34:55.793130 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0812 02:34:55.793160 (+ 30us) server_negotiation.cc:407] Connection header received
0812 02:34:55.793853 (+ 693us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:34:55.793859 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:34:55.794304 (+ 445us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:34:55.794696 (+ 392us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:34:55.796183 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:55.797191 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:55.798550 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:34:55.799103 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:34:55.799297 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:34:55.807103 (+ 7806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:34:55.807127 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:34:55.807152 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:34:55.807196 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:34:55.809431 (+ 2235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:34:55.810192 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:34:55.810201 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:34:55.810208 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:34:55.810362 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:34:55.810851 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:34:55.810856 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:34:55.810859 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:34:55.811524 (+ 665us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:34:55.811686 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0812 02:34:55.813846 (+ 2160us) server_negotiation.cc:299] Negotiation successful
0812 02:34:55.814258 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":480,"thread_start_us":90,"threads_started":1}
W20260812 02:34:55.817070 31645 leader_election.cc:341] T ca63dc3c7cdf4b0781fbfc18c270a3a5 P 649a01114e0a46628578bff8843a050a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3909f7c94d654754a9d589bb0ae09e8a (127.14.103.2:46173): Not found: Tablet not found: ca63dc3c7cdf4b0781fbfc18c270a3a5
W20260812 02:34:55.817466 31644 leader_election.cc:341] T ca63dc3c7cdf4b0781fbfc18c270a3a5 P 649a01114e0a46628578bff8843a050a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bfbbe1513aad4179abc52064d9250a9a (127.14.103.3:32843): Not found: Tablet not found: ca63dc3c7cdf4b0781fbfc18c270a3a5
I20260812 02:34:55.817600 31644 leader_election.cc:302] T ca63dc3c7cdf4b0781fbfc18c270a3a5 P 649a01114e0a46628578bff8843a050a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 649a01114e0a46628578bff8843a050a; no voters: 3909f7c94d654754a9d589bb0ae09e8a, bfbbe1513aad4179abc52064d9250a9a
I20260812 02:34:55.817948 32090 raft_consensus.cc:2726] T ca63dc3c7cdf4b0781fbfc18c270a3a5 P 649a01114e0a46628578bff8843a050a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 02:34:56.255689 32092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:56.255334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:43307 (local address 127.14.103.1:57743)
0812 02:34:56.255532 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0812 02:34:56.255600 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.62:43307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260812 02:34:56.400692 32095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:56.400241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:43307 (local address 127.14.103.2:37609)
0812 02:34:56.400533 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0812 02:34:56.400597 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.62:43307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":93,"threads_started":1}
I20260812 02:34:56.623991 32096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:56.623516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:43307 (local address 127.14.103.3:35419)
0812 02:34:56.623814 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0812 02:34:56.623866 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.62:43307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":81,"threads_started":1}
I20260812 02:34:57.257114 32097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:57.256644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:43307 (local address 127.14.103.1:35209)
0812 02:34:57.256946 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0812 02:34:57.257015 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.62:43307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":97,"threads_started":1}
I20260812 02:34:57.402366 32099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:57.401748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:43307 (local address 127.14.103.2:47055)
0812 02:34:57.402088 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0812 02:34:57.402239 (+ 151us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.62:43307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":94,"threads_started":1}
I20260812 02:34:57.625432 32100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:57.624938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:43307 (local address 127.14.103.3:54113)
0812 02:34:57.625228 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0812 02:34:57.625303 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.62:43307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":82,"threads_started":1}
I20260812 02:34:58.036368 32101 raft_consensus.cc:482] T ca63dc3c7cdf4b0781fbfc18c270a3a5 P 649a01114e0a46628578bff8843a050a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 02:34:58.036568 32101 raft_consensus.cc:504] T ca63dc3c7cdf4b0781fbfc18c270a3a5 P 649a01114e0a46628578bff8843a050a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3909f7c94d654754a9d589bb0ae09e8a" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 46173 } } peers { permanent_uuid: "bfbbe1513aad4179abc52064d9250a9a" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 32843 } } peers { permanent_uuid: "649a01114e0a46628578bff8843a050a" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 45955 } }
I20260812 02:34:58.037055 32101 leader_election.cc:288] T ca63dc3c7cdf4b0781fbfc18c270a3a5 P 649a01114e0a46628578bff8843a050a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3909f7c94d654754a9d589bb0ae09e8a (127.14.103.2:46173), bfbbe1513aad4179abc52064d9250a9a (127.14.103.3:32843)
W20260812 02:34:58.037945 31645 leader_election.cc:341] T ca63dc3c7cdf4b0781fbfc18c270a3a5 P 649a01114e0a46628578bff8843a050a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3909f7c94d654754a9d589bb0ae09e8a (127.14.103.2:46173): Not found: Tablet not found: ca63dc3c7cdf4b0781fbfc18c270a3a5
W20260812 02:34:58.038045 31644 leader_election.cc:341] T ca63dc3c7cdf4b0781fbfc18c270a3a5 P 649a01114e0a46628578bff8843a050a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bfbbe1513aad4179abc52064d9250a9a (127.14.103.3:32843): Not found: Tablet not found: ca63dc3c7cdf4b0781fbfc18c270a3a5
I20260812 02:34:58.038158 31644 leader_election.cc:302] T ca63dc3c7cdf4b0781fbfc18c270a3a5 P 649a01114e0a46628578bff8843a050a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 649a01114e0a46628578bff8843a050a; no voters: 3909f7c94d654754a9d589bb0ae09e8a, bfbbe1513aad4179abc52064d9250a9a
I20260812 02:34:58.038353 32101 raft_consensus.cc:2726] T ca63dc3c7cdf4b0781fbfc18c270a3a5 P 649a01114e0a46628578bff8843a050a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 02:34:58.258438 32102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:58.257977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:43307 (local address 127.14.103.1:58245)
0812 02:34:58.258271 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0812 02:34:58.258335 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.62:43307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":83,"threads_started":1}
I20260812 02:34:58.404291 32103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:58.403632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:43307 (local address 127.14.103.2:40353)
0812 02:34:58.404107 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0812 02:34:58.404187 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.62:43307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":83,"threads_started":1}
I20260812 02:34:58.626856 32105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:58.626358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:43307 (local address 127.14.103.3:35705)
0812 02:34:58.626676 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0812 02:34:58.626744 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.62:43307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":89,"threads_started":1}
I20260812 02:34:59.260000 32106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:59.259540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:43307 (local address 127.14.103.1:49403)
0812 02:34:59.259854 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0812 02:34:59.259906 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.62:43307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":105,"threads_started":1}
I20260812 02:34:59.405849 32107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:59.405332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:43307 (local address 127.14.103.2:48955)
0812 02:34:59.405650 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0812 02:34:59.405745 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.62:43307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":82,"threads_started":1}
I20260812 02:34:59.628329 32108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:34:59.627815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:43307 (local address 127.14.103.3:43513)
0812 02:34:59.628138 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0812 02:34:59.628230 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.62:43307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":86,"threads_started":1}
I20260812 02:35:00.261633 32109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:00.260899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:43307 (local address 127.14.103.1:47963)
0812 02:35:00.261280 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0812 02:35:00.261449 (+ 169us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.62:43307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":92,"threads_started":1}
I20260812 02:35:00.407356 32110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:00.406822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:43307 (local address 127.14.103.2:43191)
0812 02:35:00.407183 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0812 02:35:00.407253 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.62:43307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":85,"threads_started":1}
I20260812 02:35:00.629694 32111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:00.629244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:43307 (local address 127.14.103.3:60513)
0812 02:35:00.629532 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0812 02:35:00.629597 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.62:43307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":89,"threads_started":1}
I20260812 02:35:01.263309 32112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:01.262820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:43307 (local address 127.14.103.1:36787)
0812 02:35:01.263156 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0812 02:35:01.263215 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.62:43307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":77,"threads_started":1}
I20260812 02:35:01.379416 32113 raft_consensus.cc:482] T ca63dc3c7cdf4b0781fbfc18c270a3a5 P 649a01114e0a46628578bff8843a050a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 02:35:01.379565 32113 raft_consensus.cc:504] T ca63dc3c7cdf4b0781fbfc18c270a3a5 P 649a01114e0a46628578bff8843a050a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3909f7c94d654754a9d589bb0ae09e8a" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 46173 } } peers { permanent_uuid: "bfbbe1513aad4179abc52064d9250a9a" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 32843 } } peers { permanent_uuid: "649a01114e0a46628578bff8843a050a" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 45955 } }
I20260812 02:35:01.380059 32113 leader_election.cc:288] T ca63dc3c7cdf4b0781fbfc18c270a3a5 P 649a01114e0a46628578bff8843a050a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3909f7c94d654754a9d589bb0ae09e8a (127.14.103.2:46173), bfbbe1513aad4179abc52064d9250a9a (127.14.103.3:32843)
W20260812 02:35:01.381217 31644 leader_election.cc:341] T ca63dc3c7cdf4b0781fbfc18c270a3a5 P 649a01114e0a46628578bff8843a050a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bfbbe1513aad4179abc52064d9250a9a (127.14.103.3:32843): Not found: Tablet not found: ca63dc3c7cdf4b0781fbfc18c270a3a5
W20260812 02:35:01.381381 31645 leader_election.cc:341] T ca63dc3c7cdf4b0781fbfc18c270a3a5 P 649a01114e0a46628578bff8843a050a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3909f7c94d654754a9d589bb0ae09e8a (127.14.103.2:46173): Not found: Tablet not found: ca63dc3c7cdf4b0781fbfc18c270a3a5
I20260812 02:35:01.381505 31645 leader_election.cc:302] T ca63dc3c7cdf4b0781fbfc18c270a3a5 P 649a01114e0a46628578bff8843a050a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 649a01114e0a46628578bff8843a050a; no voters: 3909f7c94d654754a9d589bb0ae09e8a, bfbbe1513aad4179abc52064d9250a9a
I20260812 02:35:01.381762 32113 raft_consensus.cc:2726] T ca63dc3c7cdf4b0781fbfc18c270a3a5 P 649a01114e0a46628578bff8843a050a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 02:35:01.408722 32114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:01.408218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:43307 (local address 127.14.103.2:43677)
0812 02:35:01.408558 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0812 02:35:01.408622 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.62:43307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":102,"threads_started":1}
I20260812 02:35:01.631317 32115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:01.630675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:43307 (local address 127.14.103.3:60143)
0812 02:35:01.631151 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0812 02:35:01.631217 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.62:43307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":86,"threads_started":1}
I20260812 02:35:02.264753 32116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:02.264245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:43307 (local address 127.14.103.1:51941)
0812 02:35:02.264566 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0812 02:35:02.264634 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.62:43307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":91,"threads_started":1}
I20260812 02:35:02.410097 32117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:02.409633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:43307 (local address 127.14.103.2:57259)
0812 02:35:02.409938 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0812 02:35:02.409999 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.62:43307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":122,"threads_started":1}
I20260812 02:35:02.632697 32118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:02.632237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:43307 (local address 127.14.103.3:38475)
0812 02:35:02.632524 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0812 02:35:02.632590 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.62:43307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":79,"threads_started":1}
I20260812 02:35:03.151706 32120 raft_consensus.cc:482] T ca63dc3c7cdf4b0781fbfc18c270a3a5 P 649a01114e0a46628578bff8843a050a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 02:35:03.151845 32120 raft_consensus.cc:504] T ca63dc3c7cdf4b0781fbfc18c270a3a5 P 649a01114e0a46628578bff8843a050a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3909f7c94d654754a9d589bb0ae09e8a" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 46173 } } peers { permanent_uuid: "bfbbe1513aad4179abc52064d9250a9a" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 32843 } } peers { permanent_uuid: "649a01114e0a46628578bff8843a050a" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 45955 } }
I20260812 02:35:03.152335 32120 leader_election.cc:288] T ca63dc3c7cdf4b0781fbfc18c270a3a5 P 649a01114e0a46628578bff8843a050a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3909f7c94d654754a9d589bb0ae09e8a (127.14.103.2:46173), bfbbe1513aad4179abc52064d9250a9a (127.14.103.3:32843)
W20260812 02:35:03.153170 31644 leader_election.cc:341] T ca63dc3c7cdf4b0781fbfc18c270a3a5 P 649a01114e0a46628578bff8843a050a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bfbbe1513aad4179abc52064d9250a9a (127.14.103.3:32843): Not found: Tablet not found: ca63dc3c7cdf4b0781fbfc18c270a3a5
W20260812 02:35:03.153308 31645 leader_election.cc:341] T ca63dc3c7cdf4b0781fbfc18c270a3a5 P 649a01114e0a46628578bff8843a050a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3909f7c94d654754a9d589bb0ae09e8a (127.14.103.2:46173): Not found: Tablet not found: ca63dc3c7cdf4b0781fbfc18c270a3a5
I20260812 02:35:03.153405 31645 leader_election.cc:302] T ca63dc3c7cdf4b0781fbfc18c270a3a5 P 649a01114e0a46628578bff8843a050a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 649a01114e0a46628578bff8843a050a; no voters: 3909f7c94d654754a9d589bb0ae09e8a, bfbbe1513aad4179abc52064d9250a9a
I20260812 02:35:03.153551 32120 raft_consensus.cc:2726] T ca63dc3c7cdf4b0781fbfc18c270a3a5 P 649a01114e0a46628578bff8843a050a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 02:35:03.266355 32121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:03.265830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:43307 (local address 127.14.103.1:55993)
0812 02:35:03.266205 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0812 02:35:03.266259 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.62:43307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":83,"threads_started":1}
I20260812 02:35:03.411689 32122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:03.411078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:43307 (local address 127.14.103.2:33821)
0812 02:35:03.411473 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0812 02:35:03.411543 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.62:43307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":88,"threads_started":1}
I20260812 02:35:03.634152 32123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:03.633683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:43307 (local address 127.14.103.3:33409)
0812 02:35:03.633975 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0812 02:35:03.634044 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.62:43307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":88,"threads_started":1}
I20260812 02:35:04.268280 32124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:04.267529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:43307 (local address 127.14.103.1:52931)
0812 02:35:04.267854 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0812 02:35:04.267969 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.62:43307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":93,"threads_started":1}
I20260812 02:35:04.413184 32125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:04.412640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:43307 (local address 127.14.103.2:43175)
0812 02:35:04.412987 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0812 02:35:04.413062 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.62:43307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":85,"threads_started":1}
I20260812 02:35:04.635813 32127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:04.635352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:43307 (local address 127.14.103.3:39733)
0812 02:35:04.635629 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0812 02:35:04.635689 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.62:43307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":77,"threads_started":1}
I20260812 02:35:05.269825 32128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:05.269352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:43307 (local address 127.14.103.1:57427)
0812 02:35:05.269659 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0812 02:35:05.269727 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.62:43307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":87,"threads_started":1}
I20260812 02:35:05.415093 32129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:05.414460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:43307 (local address 127.14.103.2:38919)
0812 02:35:05.414874 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0812 02:35:05.414989 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.62:43307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":194,"threads_started":1}
I20260812 02:35:05.637512 32130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:05.636973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:43307 (local address 127.14.103.3:36555)
0812 02:35:05.637304 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0812 02:35:05.637402 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.62:43307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":144,"threads_started":1}
I20260812 02:35:06.271430 32131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:06.270877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:43307 (local address 127.14.103.1:33747)
0812 02:35:06.271244 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0812 02:35:06.271308 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.62:43307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":155,"threads_started":1}
I20260812 02:35:06.416550 32132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:06.416037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:43307 (local address 127.14.103.2:56539)
0812 02:35:06.416371 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0812 02:35:06.416450 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.62:43307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":111,"threads_started":1}
I20260812 02:35:06.639506 32133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:06.638923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:43307 (local address 127.14.103.3:44783)
0812 02:35:06.639279 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0812 02:35:06.639350 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.62:43307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":129,"threads_started":1}
I20260812 02:35:07.272864 32134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:07.272268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:43307 (local address 127.14.103.1:46045)
0812 02:35:07.272602 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0812 02:35:07.272742 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.62:43307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":79,"threads_started":1}
I20260812 02:35:07.417902 32136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:07.417425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:43307 (local address 127.14.103.2:44833)
0812 02:35:07.417746 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0812 02:35:07.417805 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.62:43307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":122,"threads_started":1}
I20260812 02:35:07.641455 32137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:07.640878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:43307 (local address 127.14.103.3:42015)
0812 02:35:07.641222 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0812 02:35:07.641302 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.62:43307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":111,"threads_started":1}
I20260812 02:35:08.274385 32139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:08.273715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:43307 (local address 127.14.103.1:60733)
0812 02:35:08.274072 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0812 02:35:08.274146 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.62:43307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":120,"threads_started":1}
I20260812 02:35:08.419931 32140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:08.419212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:43307 (local address 127.14.103.2:60395)
0812 02:35:08.419709 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0812 02:35:08.419787 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.62:43307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":318,"threads_started":1}
I20260812 02:35:08.643074 32141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:08.642537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:43307 (local address 127.14.103.3:59625)
0812 02:35:08.642874 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0812 02:35:08.642943 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.62:43307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":106,"threads_started":1}
I20260812 02:35:08.938668 32142 raft_consensus.cc:482] T ca63dc3c7cdf4b0781fbfc18c270a3a5 P 649a01114e0a46628578bff8843a050a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 02:35:08.938866 32142 raft_consensus.cc:504] T ca63dc3c7cdf4b0781fbfc18c270a3a5 P 649a01114e0a46628578bff8843a050a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3909f7c94d654754a9d589bb0ae09e8a" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 46173 } } peers { permanent_uuid: "bfbbe1513aad4179abc52064d9250a9a" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 32843 } } peers { permanent_uuid: "649a01114e0a46628578bff8843a050a" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 45955 } }
I20260812 02:35:08.939357 32142 leader_election.cc:288] T ca63dc3c7cdf4b0781fbfc18c270a3a5 P 649a01114e0a46628578bff8843a050a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3909f7c94d654754a9d589bb0ae09e8a (127.14.103.2:46173), bfbbe1513aad4179abc52064d9250a9a (127.14.103.3:32843)
W20260812 02:35:08.940582 31645 leader_election.cc:341] T ca63dc3c7cdf4b0781fbfc18c270a3a5 P 649a01114e0a46628578bff8843a050a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3909f7c94d654754a9d589bb0ae09e8a (127.14.103.2:46173): Not found: Tablet not found: ca63dc3c7cdf4b0781fbfc18c270a3a5
W20260812 02:35:08.940894 31644 leader_election.cc:341] T ca63dc3c7cdf4b0781fbfc18c270a3a5 P 649a01114e0a46628578bff8843a050a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bfbbe1513aad4179abc52064d9250a9a (127.14.103.3:32843): Not found: Tablet not found: ca63dc3c7cdf4b0781fbfc18c270a3a5
I20260812 02:35:08.940973 31644 leader_election.cc:302] T ca63dc3c7cdf4b0781fbfc18c270a3a5 P 649a01114e0a46628578bff8843a050a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 649a01114e0a46628578bff8843a050a; no voters: 3909f7c94d654754a9d589bb0ae09e8a, bfbbe1513aad4179abc52064d9250a9a
I20260812 02:35:08.941244 32142 raft_consensus.cc:2726] T ca63dc3c7cdf4b0781fbfc18c270a3a5 P 649a01114e0a46628578bff8843a050a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 02:35:09.275974 32143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:09.275508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:43307 (local address 127.14.103.1:48465)
0812 02:35:09.275813 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0812 02:35:09.275870 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.62:43307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":96,"threads_started":1}
I20260812 02:35:09.421417 32144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:09.420974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:43307 (local address 127.14.103.2:52159)
0812 02:35:09.421262 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0812 02:35:09.421319 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.62:43307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":78,"threads_started":1}
I20260812 02:35:09.622298 14748 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbep4M4/build/debug/bin/kudu with pid 31586
I20260812 02:35:09.634333 14748 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbep4M4/build/debug/bin/kudu with pid 31779
I20260812 02:35:09.645108 32145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:09.644522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:43307 (local address 127.14.103.3:60685)
0812 02:35:09.644861 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0812 02:35:09.644954 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.62:43307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":88,"threads_started":1}
I20260812 02:35:09.647197 14748 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbep4M4/build/debug/bin/kudu with pid 31933
2026-08-12T02:35:09Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19568 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 02:35:09 dist-test-slave-2w3w krb5kdc[32148](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 02:35:09 dist-test-slave-2w3w krb5kdc[32148](info): set up 2 sockets
Aug 12 02:35:09 dist-test-slave-2w3w krb5kdc[32148](info): commencing operation
krb5kdc: starting...
W20260812 02:35:10.813074 14748 mini_kdc.cc:121] Time spent starting KDC: real 1.136s user 0.002s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/krb5kdc/test-user.keytab.
Aug 12 02:35:10 dist-test-slave-2w3w krb5kdc[32148](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502110, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T02:35:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T02:35:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.103.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.103.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.103.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.62.keytab.
Entry for principal kudu/127.14.103.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.62.keytab.
I20260812 02:35:10.964079 14748 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbep4M4/build/debug/bin/kudu
/tmp/dist-test-taskbep4M4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.103.62:36667
--webserver_interface=127.14.103.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.62.keytab
--principal=kudu/127.14.103.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.103.20:45573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.103.62:36667
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:35:11.090638 32166 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:35:11.091094 32166 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:35:11.091218 32166 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:35:11.095573 32166 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260812 02:35:11.095692 32166 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260812 02:35:11.095721 32166 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260812 02:35:11.095744 32166 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:35:11.095789 32166 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:35:11.095830 32166 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:35:11.095878 32166 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:35:11.095916 32166 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 02:35:11.102147 32166 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.103.20:45573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.14.103.62:36667
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.62.keytab
--principal=kudu/127.14.103.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.103.62:36667
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.103.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32166
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:35:11.103466 32166 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:35:11.104605 32166 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:35:11.114780 32176 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:35:11.114780 32172 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:35:11.114897 32166 server_base.cc:1034] running on GCE node
W20260812 02:35:11.114738 32171 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:35:11.115933 32166 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:35:11.117193 32166 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:35:11.118429 32166 hybrid_clock.cc:648] HybridClock initialized: now 1786502111118410 us; error 44 us; skew 500 ppm
Aug 12 02:35:11 dist-test-slave-2w3w krb5kdc[32148](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502111, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:35:11.123498 32166 init.cc:377] Logged in from keytab as kudu/127.14.103.62@KRBTEST.COM (short username kudu)
I20260812 02:35:11.124636 32166 webserver.cc:460] Webserver started at http://127.14.103.62:34833/ using document root <none> and password file <none>
I20260812 02:35:11.125309 32166 fs_manager.cc:359] Metadata directory not provided
I20260812 02:35:11.125388 32166 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:35:11.125638 32166 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:35:11.128237 32166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/minicluster-data/master-0/data/instance:
uuid: "1806e8fbbfe747018262028788ff8fcf"
format_stamp: "Formatted at 2026-08-12 02:35:11 on dist-test-slave-2w3w"
server_key: "31908b9dfc9c08e00ad8c34bfd89f6ae"
server_key_iv: "6c2b36b4aacae94d61c9d53340224cd2"
server_key_version: "encryptionkey@0"
I20260812 02:35:11.128958 32166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/minicluster-data/master-0/wal/instance:
uuid: "1806e8fbbfe747018262028788ff8fcf"
format_stamp: "Formatted at 2026-08-12 02:35:11 on dist-test-slave-2w3w"
server_key: "31908b9dfc9c08e00ad8c34bfd89f6ae"
server_key_iv: "6c2b36b4aacae94d61c9d53340224cd2"
server_key_version: "encryptionkey@0"
I20260812 02:35:11.133718 32166 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260812 02:35:11.138023 32183 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:35:11.139830 32166 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260812 02:35:11.140056 32166 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/minicluster-data/master-0/data,/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/minicluster-data/master-0/wal
uuid: "1806e8fbbfe747018262028788ff8fcf"
format_stamp: "Formatted at 2026-08-12 02:35:11 on dist-test-slave-2w3w"
server_key: "31908b9dfc9c08e00ad8c34bfd89f6ae"
server_key_iv: "6c2b36b4aacae94d61c9d53340224cd2"
server_key_version: "encryptionkey@0"
I20260812 02:35:11.140213 32166 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:35:11.154847 32166 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:35:11.159534 32166 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:35:11.160046 32166 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:35:11.169235 32166 rpc_server.cc:307] RPC server started. Bound to: 127.14.103.62:36667
I20260812 02:35:11.169564 32242 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.103.62:36667 every 8 connection(s)
I20260812 02:35:11.170356 32166 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/minicluster-data/master-0/data/info.pb
I20260812 02:35:11.173566 32243 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:35:11.173969 14748 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbep4M4/build/debug/bin/kudu as pid 32166
I20260812 02:35:11.174134 14748 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/minicluster-data/master-0/wal/instance
I20260812 02:35:11.174486 14748 external_mini_cluster.cc:1428] Setting key 1bbaa1b7d6b622ca20f2e961d7a3dc84
I20260812 02:35:11.182385 32243 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1806e8fbbfe747018262028788ff8fcf: Bootstrap starting.
I20260812 02:35:11.185393 32243 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1806e8fbbfe747018262028788ff8fcf: Neither blocks nor log segments found. Creating new log.
I20260812 02:35:11.186241 32243 log.cc:824] T 00000000000000000000000000000000 P 1806e8fbbfe747018262028788ff8fcf: Log is configured to *not* fsync() on all Append() calls
Aug 12 02:35:11 dist-test-slave-2w3w krb5kdc[32148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502110, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.103.62@KRBTEST.COM
I20260812 02:35:11.188978 32243 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1806e8fbbfe747018262028788ff8fcf: No bootstrap required, opened a new log
I20260812 02:35:11.192714 32246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:11.176342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37280 (local address 127.14.103.62:36667)
0812 02:35:11.176879 (+ 537us) server_negotiation.cc:206] Beginning negotiation
0812 02:35:11.176898 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0812 02:35:11.176946 (+ 48us) server_negotiation.cc:407] Connection header received
0812 02:35:11.177724 (+ 778us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:35:11.177757 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:35:11.178241 (+ 484us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:35:11.178762 (+ 521us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:35:11.179669 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:35:11.181324 (+ 1655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:35:11.183298 (+ 1974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:35:11.184087 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:35:11.184280 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:35:11.187242 (+ 2962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:35:11.187251 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:35:11.187273 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:35:11.187309 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:35:11.189239 (+ 1930us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:35:11.189830 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:35:11.189835 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:35:11.189840 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:35:11.189988 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:35:11.190360 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:35:11.190363 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:35:11.190365 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:35:11.190827 (+ 462us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:35:11.191120 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0812 02:35:11.191465 (+ 345us) server_negotiation.cc:299] Negotiation successful
0812 02:35:11.191874 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":138,"threads_started":1}
I20260812 02:35:11.193512 32243 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1806e8fbbfe747018262028788ff8fcf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1806e8fbbfe747018262028788ff8fcf" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 36667 } }
I20260812 02:35:11.193936 32243 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1806e8fbbfe747018262028788ff8fcf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:35:11.194032 32243 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1806e8fbbfe747018262028788ff8fcf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1806e8fbbfe747018262028788ff8fcf, State: Initialized, Role: FOLLOWER
I20260812 02:35:11.194510 32243 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1806e8fbbfe747018262028788ff8fcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1806e8fbbfe747018262028788ff8fcf" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 36667 } }
I20260812 02:35:11.194691 32243 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1806e8fbbfe747018262028788ff8fcf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 02:35:11.194802 32243 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1806e8fbbfe747018262028788ff8fcf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 02:35:11.194941 32243 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1806e8fbbfe747018262028788ff8fcf [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:35:11.196278 32243 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1806e8fbbfe747018262028788ff8fcf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1806e8fbbfe747018262028788ff8fcf" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 36667 } }
I20260812 02:35:11.196751 32243 leader_election.cc:302] T 00000000000000000000000000000000 P 1806e8fbbfe747018262028788ff8fcf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1806e8fbbfe747018262028788ff8fcf; no voters:
I20260812 02:35:11.197367 32243 leader_election.cc:288] T 00000000000000000000000000000000 P 1806e8fbbfe747018262028788ff8fcf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 02:35:11.197379 32249 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1806e8fbbfe747018262028788ff8fcf [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:35:11.197729 32249 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1806e8fbbfe747018262028788ff8fcf [term 1 LEADER]: Becoming Leader. State: Replica: 1806e8fbbfe747018262028788ff8fcf, State: Running, Role: LEADER
I20260812 02:35:11.198123 32249 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1806e8fbbfe747018262028788ff8fcf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1806e8fbbfe747018262028788ff8fcf" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 36667 } }
I20260812 02:35:11.198803 32243 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1806e8fbbfe747018262028788ff8fcf [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:35:11.200302 32252 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1806e8fbbfe747018262028788ff8fcf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1806e8fbbfe747018262028788ff8fcf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1806e8fbbfe747018262028788ff8fcf" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 36667 } } }
I20260812 02:35:11.200502 32252 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1806e8fbbfe747018262028788ff8fcf [sys.catalog]: This master's current role is: LEADER
I20260812 02:35:11.200412 32251 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1806e8fbbfe747018262028788ff8fcf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1806e8fbbfe747018262028788ff8fcf. Latest consensus state: current_term: 1 leader_uuid: "1806e8fbbfe747018262028788ff8fcf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1806e8fbbfe747018262028788ff8fcf" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 36667 } } }
I20260812 02:35:11.200737 32251 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1806e8fbbfe747018262028788ff8fcf [sys.catalog]: This master's current role is: LEADER
I20260812 02:35:11.200981 32260 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 02:35:11.204602 32260 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 02:35:11.210812 32260 catalog_manager.cc:1347] Generated new cluster ID: 4bc06eec6ed3497e94bf64b1bc9c3590
I20260812 02:35:11.210916 32260 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 02:35:11.230473 32260 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 02:35:11.231956 32260 catalog_manager.cc:1504] Loading token signing keys...
I20260812 02:35:11.245460 32260 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1806e8fbbfe747018262028788ff8fcf: Generated new TSK 0
I20260812 02:35:11.246503 32260 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.103.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.103.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.103.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.1.keytab.
Entry for principal kudu/127.14.103.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.1.keytab.
I20260812 02:35:11.295657 14748 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbep4M4/build/debug/bin/kudu
/tmp/dist-test-taskbep4M4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.103.1:0
--local_ip_for_outbound_sockets=127.14.103.1
--webserver_interface=127.14.103.1
--webserver_port=0
--tserver_master_addrs=127.14.103.62:36667
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.1.keytab
--principal=kudu/127.14.103.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.103.20:45573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:35:11.420047 32273 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:35:11.420619 32273 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:35:11.420722 32273 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:35:11.425976 32273 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:35:11.426122 32273 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:35:11.426235 32273 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.103.1
I20260812 02:35:11.432494 32273 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.103.20:45573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.1.keytab
--principal=kudu/127.14.103.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.103.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.103.1
--webserver_port=0
--tserver_master_addrs=127.14.103.62:36667
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32273
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.103.1
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:35:11.433774 32273 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:35:11.434922 32273 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:35:11.446559 32279 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:35:11.446700 32273 server_base.cc:1034] running on GCE node
W20260812 02:35:11.446722 32280 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:35:11.446831 32282 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:35:11.447556 32273 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:35:11.448313 32273 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:35:11.449548 32273 hybrid_clock.cc:648] HybridClock initialized: now 1786502111449508 us; error 55 us; skew 500 ppm
Aug 12 02:35:11 dist-test-slave-2w3w krb5kdc[32148](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502111, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:35:11.453109 32273 init.cc:377] Logged in from keytab as kudu/127.14.103.1@KRBTEST.COM (short username kudu)
I20260812 02:35:11.454221 32273 webserver.cc:460] Webserver started at http://127.14.103.1:35679/ using document root <none> and password file <none>
I20260812 02:35:11.455683 32273 fs_manager.cc:359] Metadata directory not provided
I20260812 02:35:11.455782 32273 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:35:11.455991 32273 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:35:11.458319 32273 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/minicluster-data/ts-0/data/instance:
uuid: "eb3490e69b2a4d72947384cae82a0d4e"
format_stamp: "Formatted at 2026-08-12 02:35:11 on dist-test-slave-2w3w"
server_key: "c676407f995241b3a413e8c7aabc506c"
server_key_iv: "e3fd61796ec1f02c47b288f367d1e9f9"
server_key_version: "encryptionkey@0"
I20260812 02:35:11.458937 32273 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/minicluster-data/ts-0/wal/instance:
uuid: "eb3490e69b2a4d72947384cae82a0d4e"
format_stamp: "Formatted at 2026-08-12 02:35:11 on dist-test-slave-2w3w"
server_key: "c676407f995241b3a413e8c7aabc506c"
server_key_iv: "e3fd61796ec1f02c47b288f367d1e9f9"
server_key_version: "encryptionkey@0"
I20260812 02:35:11.463441 32273 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260812 02:35:11.466423 32291 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:35:11.468050 32273 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 02:35:11.468192 32273 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/minicluster-data/ts-0/wal
uuid: "eb3490e69b2a4d72947384cae82a0d4e"
format_stamp: "Formatted at 2026-08-12 02:35:11 on dist-test-slave-2w3w"
server_key: "c676407f995241b3a413e8c7aabc506c"
server_key_iv: "e3fd61796ec1f02c47b288f367d1e9f9"
server_key_version: "encryptionkey@0"
I20260812 02:35:11.468389 32273 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:35:11.494917 32273 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:35:11.498811 32273 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:35:11.499138 32273 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:35:11.499842 32273 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:35:11.500836 32273 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:35:11.500910 32273 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:35:11.500986 32273 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:35:11.501019 32273 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:35:11.511422 32273 rpc_server.cc:307] RPC server started. Bound to: 127.14.103.1:37887
I20260812 02:35:11.511462 32417 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.103.1:37887 every 8 connection(s)
I20260812 02:35:11.512460 32273 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/minicluster-data/ts-0/data/info.pb
I20260812 02:35:11.515391 14748 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbep4M4/build/debug/bin/kudu as pid 32273
I20260812 02:35:11.515606 14748 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/minicluster-data/ts-0/wal/instance
I20260812 02:35:11.515974 14748 external_mini_cluster.cc:1428] Setting key ec5c6a55b3786b998e39c2ed80967a46
Aug 12 02:35:11 dist-test-slave-2w3w krb5kdc[32148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502111, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.1@KRBTEST.COM for kudu/127.14.103.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.14.103.2@KRBTEST.COM; defaulting to no policy
I20260812 02:35:11.530594 32246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:11.514688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.1:59159 (local address 127.14.103.62:36667)
0812 02:35:11.514903 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0812 02:35:11.514907 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:35:11.516286 (+ 1379us) server_negotiation.cc:407] Connection header received
0812 02:35:11.517442 (+ 1156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:35:11.517449 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:35:11.517515 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:35:11.517615 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:35:11.518953 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:35:11.519981 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:35:11.522021 (+ 2040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:35:11.522306 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:35:11.522407 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:35:11.526105 (+ 3698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:35:11.526109 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:35:11.526115 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:35:11.526149 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:35:11.528111 (+ 1962us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:35:11.528941 (+ 830us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:35:11.528945 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:35:11.528946 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:35:11.529036 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:35:11.529394 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:35:11.529396 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:35:11.529397 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:35:11.529593 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:35:11.529670 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0812 02:35:11.530323 (+ 653us) server_negotiation.cc:299] Negotiation successful
0812 02:35:11.530459 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122}
I20260812 02:35:11.531646 32420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:11.515138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:36667 (local address 127.14.103.1:59159)
0812 02:35:11.516048 (+ 910us) negotiation.cc:107] Waiting for socket to connect
0812 02:35:11.516121 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0812 02:35:11.517138 (+ 1017us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:35:11.517829 (+ 691us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:35:11.517837 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:35:11.518242 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:35:11.518782 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:35:11.518796 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:35:11.520177 (+ 1381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:35:11.520183 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:35:11.521843 (+ 1660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:35:11.521860 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:35:11.522503 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:35:11.522510 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:35:11.522763 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:35:11.523516 (+ 753us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:35:11.523544 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:35:11.525733 (+ 2189us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:35:11.528269 (+ 2536us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:35:11.528277 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:35:11.528298 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:35:11.528728 (+ 430us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:35:11.529141 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:35:11.529144 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:35:11.529145 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:35:11.529306 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:35:11.529690 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:35:11.529695 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:35:11.530065 (+ 370us) client_negotiation.cc:769] Sending connection context
0812 02:35:11.530246 (+ 181us) client_negotiation.cc:240] Negotiation successful
0812 02:35:11.530550 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":654,"thread_start_us":147,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.103.2@KRBTEST.COM" created.
I20260812 02:35:11.533457 32418 heartbeater.cc:347] Connected to a master server at 127.14.103.62:36667
I20260812 02:35:11.533761 32418 heartbeater.cc:464] Registering TS with master...
I20260812 02:35:11.534420 32418 heartbeater.cc:511] Master 127.14.103.62:36667 requested a full tablet report, sending...
I20260812 02:35:11.536360 32203 ts_manager.cc:194] Registered new tserver with Master: eb3490e69b2a4d72947384cae82a0d4e (127.14.103.1:37887)
I20260812 02:35:11.538064 32203 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.103.1@KRBTEST.COM'} at 127.14.103.1:59159
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.103.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.2.keytab.
Entry for principal kudu/127.14.103.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.2.keytab.
I20260812 02:35:11.556351 14748 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbep4M4/build/debug/bin/kudu
/tmp/dist-test-taskbep4M4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.103.2:0
--local_ip_for_outbound_sockets=127.14.103.2
--webserver_interface=127.14.103.2
--webserver_port=0
--tserver_master_addrs=127.14.103.62:36667
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.2.keytab
--principal=kudu/127.14.103.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.103.20:45573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:35:11.678800 32423 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:35:11.679229 32423 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:35:11.679283 32423 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:35:11.683665 32423 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:35:11.683744 32423 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:35:11.683811 32423 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.103.2
I20260812 02:35:11.689790 32423 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.103.20:45573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.2.keytab
--principal=kudu/127.14.103.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.103.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.103.2
--webserver_port=0
--tserver_master_addrs=127.14.103.62:36667
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32423
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.103.2
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:35:11.690937 32423 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:35:11.692076 32423 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:35:11.704376 32432 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:35:11.704403 32429 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:35:11.704527 32430 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:35:11.704726 32423 server_base.cc:1034] running on GCE node
I20260812 02:35:11.705762 32423 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:35:11.706810 32423 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:35:11.708045 32423 hybrid_clock.cc:648] HybridClock initialized: now 1786502111708014 us; error 48 us; skew 500 ppm
Aug 12 02:35:11 dist-test-slave-2w3w krb5kdc[32148](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502111, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:35:11.711565 32423 init.cc:377] Logged in from keytab as kudu/127.14.103.2@KRBTEST.COM (short username kudu)
I20260812 02:35:11.712700 32423 webserver.cc:460] Webserver started at http://127.14.103.2:44487/ using document root <none> and password file <none>
I20260812 02:35:11.713338 32423 fs_manager.cc:359] Metadata directory not provided
I20260812 02:35:11.713440 32423 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:35:11.713696 32423 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:35:11.716049 32423 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/minicluster-data/ts-1/data/instance:
uuid: "4c0b137c9b9044e49094207590ce977e"
format_stamp: "Formatted at 2026-08-12 02:35:11 on dist-test-slave-2w3w"
server_key: "f4f69eb05b1f444046bfdcdd6e2e1ce8"
server_key_iv: "793d2d7b936db6cd2822f67e83dc527e"
server_key_version: "encryptionkey@0"
I20260812 02:35:11.716684 32423 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/minicluster-data/ts-1/wal/instance:
uuid: "4c0b137c9b9044e49094207590ce977e"
format_stamp: "Formatted at 2026-08-12 02:35:11 on dist-test-slave-2w3w"
server_key: "f4f69eb05b1f444046bfdcdd6e2e1ce8"
server_key_iv: "793d2d7b936db6cd2822f67e83dc527e"
server_key_version: "encryptionkey@0"
I20260812 02:35:11.721102 32423 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260812 02:35:11.723992 32440 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:35:11.725379 32423 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260812 02:35:11.725555 32423 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/minicluster-data/ts-1/wal
uuid: "4c0b137c9b9044e49094207590ce977e"
format_stamp: "Formatted at 2026-08-12 02:35:11 on dist-test-slave-2w3w"
server_key: "f4f69eb05b1f444046bfdcdd6e2e1ce8"
server_key_iv: "793d2d7b936db6cd2822f67e83dc527e"
server_key_version: "encryptionkey@0"
I20260812 02:35:11.725694 32423 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:35:11.744268 32423 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:35:11.748330 32423 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:35:11.748579 32423 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:35:11.749243 32423 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:35:11.750139 32423 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:35:11.750213 32423 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:35:11.750288 32423 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:35:11.750340 32423 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:35:11.760210 32423 rpc_server.cc:307] RPC server started. Bound to: 127.14.103.2:45799
I20260812 02:35:11.760228 32584 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.103.2:45799 every 8 connection(s)
I20260812 02:35:11.761230 32423 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/minicluster-data/ts-1/data/info.pb
I20260812 02:35:11.765347 14748 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbep4M4/build/debug/bin/kudu as pid 32423
I20260812 02:35:11.765659 14748 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/minicluster-data/ts-1/wal/instance
I20260812 02:35:11.766076 14748 external_mini_cluster.cc:1428] Setting key dedcb49a71356e6a6c95f6f7440436c2
Aug 12 02:35:11 dist-test-slave-2w3w krb5kdc[32148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502111, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.2@KRBTEST.COM for kudu/127.14.103.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.14.103.3@KRBTEST.COM; defaulting to no policy
I20260812 02:35:11.781313 32246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:11.764039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.2:43843 (local address 127.14.103.62:36667)
0812 02:35:11.764202 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0812 02:35:11.764206 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:35:11.765429 (+ 1223us) server_negotiation.cc:407] Connection header received
0812 02:35:11.767274 (+ 1845us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:35:11.767279 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:35:11.767338 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:35:11.767405 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:35:11.769600 (+ 2195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:35:11.770425 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:35:11.771893 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:35:11.772159 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:35:11.772288 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:35:11.775808 (+ 3520us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:35:11.775814 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:35:11.775817 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:35:11.775856 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:35:11.777843 (+ 1987us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:35:11.778522 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:35:11.778529 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:35:11.778531 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:35:11.778652 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:35:11.779448 (+ 796us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:35:11.779452 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:35:11.779454 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:35:11.779799 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:35:11.779995 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0812 02:35:11.780857 (+ 862us) server_negotiation.cc:299] Negotiation successful
0812 02:35:11.781079 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260812 02:35:11.781958 32587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:11.764386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:36667 (local address 127.14.103.2:43843)
0812 02:35:11.765241 (+ 855us) negotiation.cc:107] Waiting for socket to connect
0812 02:35:11.765289 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0812 02:35:11.766991 (+ 1702us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:35:11.767990 (+ 999us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:35:11.768003 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:35:11.768677 (+ 674us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:35:11.769411 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:35:11.769431 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:35:11.770630 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:35:11.770636 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:35:11.771727 (+ 1091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:35:11.771740 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:35:11.772356 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:35:11.772361 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:35:11.772586 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:35:11.773361 (+ 775us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:35:11.773415 (+ 54us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:35:11.775517 (+ 2102us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:35:11.778027 (+ 2510us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:35:11.778033 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:35:11.778050 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:35:11.778375 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:35:11.778803 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:35:11.778806 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:35:11.778808 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:35:11.779097 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:35:11.780002 (+ 905us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:35:11.780008 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:35:11.780308 (+ 300us) client_negotiation.cc:769] Sending connection context
0812 02:35:11.780494 (+ 186us) client_negotiation.cc:240] Negotiation successful
0812 02:35:11.780948 (+ 454us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":624,"thread_start_us":154,"threads_started":1}
I20260812 02:35:11.783535 32585 heartbeater.cc:347] Connected to a master server at 127.14.103.62:36667
I20260812 02:35:11.783886 32585 heartbeater.cc:464] Registering TS with master...
I20260812 02:35:11.784536 32585 heartbeater.cc:511] Master 127.14.103.62:36667 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.103.3@KRBTEST.COM" created.
I20260812 02:35:11.786305 32203 ts_manager.cc:194] Registered new tserver with Master: 4c0b137c9b9044e49094207590ce977e (127.14.103.2:45799)
I20260812 02:35:11.787245 32203 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.103.2@KRBTEST.COM'} at 127.14.103.2:43843
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.103.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.3.keytab.
Entry for principal kudu/127.14.103.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.3.keytab.
I20260812 02:35:11.808648 14748 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbep4M4/build/debug/bin/kudu
/tmp/dist-test-taskbep4M4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.103.3:0
--local_ip_for_outbound_sockets=127.14.103.3
--webserver_interface=127.14.103.3
--webserver_port=0
--tserver_master_addrs=127.14.103.62:36667
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.3.keytab
--principal=kudu/127.14.103.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.103.20:45573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:35:11.933821 32592 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:35:11.934342 32592 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:35:11.934480 32592 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:35:11.939045 32592 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:35:11.939168 32592 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:35:11.939273 32592 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.103.3
I20260812 02:35:11.945112 32592 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.103.20:45573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.3.keytab
--principal=kudu/127.14.103.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.103.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.103.3
--webserver_port=0
--tserver_master_addrs=127.14.103.62:36667
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32592
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.103.3
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:35:11.946412 32592 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:35:11.947547 32592 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:35:11.958046 32598 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:35:11.958072 32597 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:35:11.958046 32600 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:35:11.958231 32592 server_base.cc:1034] running on GCE node
I20260812 02:35:11.958803 32592 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:35:11.959534 32592 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:35:11.960731 32592 hybrid_clock.cc:648] HybridClock initialized: now 1786502111960716 us; error 28 us; skew 500 ppm
Aug 12 02:35:11 dist-test-slave-2w3w krb5kdc[32148](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502111, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:35:11.965644 32592 init.cc:377] Logged in from keytab as kudu/127.14.103.3@KRBTEST.COM (short username kudu)
I20260812 02:35:11.966820 32592 webserver.cc:460] Webserver started at http://127.14.103.3:44141/ using document root <none> and password file <none>
I20260812 02:35:11.967550 32592 fs_manager.cc:359] Metadata directory not provided
I20260812 02:35:11.967630 32592 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:35:11.967890 32592 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:35:11.970396 32592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/minicluster-data/ts-2/data/instance:
uuid: "e8395e0332674f379f8c508b96e4e0c0"
format_stamp: "Formatted at 2026-08-12 02:35:11 on dist-test-slave-2w3w"
server_key: "985211a092ad46521774423ece2bee0a"
server_key_iv: "a128ee492a442ed5171334ed7cbc8269"
server_key_version: "encryptionkey@0"
I20260812 02:35:11.971140 32592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/minicluster-data/ts-2/wal/instance:
uuid: "e8395e0332674f379f8c508b96e4e0c0"
format_stamp: "Formatted at 2026-08-12 02:35:11 on dist-test-slave-2w3w"
server_key: "985211a092ad46521774423ece2bee0a"
server_key_iv: "a128ee492a442ed5171334ed7cbc8269"
server_key_version: "encryptionkey@0"
I20260812 02:35:11.976179 32592 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260812 02:35:11.979104 32608 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:35:11.980571 32592 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 02:35:11.980756 32592 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/minicluster-data/ts-2/wal
uuid: "e8395e0332674f379f8c508b96e4e0c0"
format_stamp: "Formatted at 2026-08-12 02:35:11 on dist-test-slave-2w3w"
server_key: "985211a092ad46521774423ece2bee0a"
server_key_iv: "a128ee492a442ed5171334ed7cbc8269"
server_key_version: "encryptionkey@0"
I20260812 02:35:11.980903 32592 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:35:12.009553 32592 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:35:12.013942 32592 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:35:12.014184 32592 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:35:12.014875 32592 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:35:12.015830 32592 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:35:12.015905 32592 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:35:12.015981 32592 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:35:12.016023 32592 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:35:12.026850 32592 rpc_server.cc:307] RPC server started. Bound to: 127.14.103.3:33023
I20260812 02:35:12.026911 32741 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.103.3:33023 every 8 connection(s)
I20260812 02:35:12.027943 32592 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/minicluster-data/ts-2/data/info.pb
I20260812 02:35:12.038012 14748 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbep4M4/build/debug/bin/kudu as pid 32592
I20260812 02:35:12.038154 14748 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/minicluster-data/ts-2/wal/instance
I20260812 02:35:12.038450 14748 external_mini_cluster.cc:1428] Setting key b2783b8ab8876c783d5e6814e401c420
Aug 12 02:35:12 dist-test-slave-2w3w krb5kdc[32148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502111, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.3@KRBTEST.COM for kudu/127.14.103.62@KRBTEST.COM
I20260812 02:35:12.043973 32246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:12.029978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.3:52677 (local address 127.14.103.62:36667)
0812 02:35:12.030140 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0812 02:35:12.030143 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 02:35:12.031519 (+ 1376us) server_negotiation.cc:407] Connection header received
0812 02:35:12.032515 (+ 996us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:35:12.032519 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:35:12.032585 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:35:12.032703 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:35:12.034042 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:35:12.034619 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:35:12.035956 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:35:12.036194 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:35:12.036320 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:35:12.039487 (+ 3167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:35:12.039492 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:35:12.039496 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:35:12.039531 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:35:12.041352 (+ 1821us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:35:12.042010 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:35:12.042013 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:35:12.042014 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:35:12.042113 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:35:12.042654 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:35:12.042657 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:35:12.042658 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:35:12.042885 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:35:12.043100 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0812 02:35:12.043693 (+ 593us) server_negotiation.cc:299] Negotiation successful
0812 02:35:12.043816 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260812 02:35:12.044883 32744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:12.030443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:36667 (local address 127.14.103.3:52677)
0812 02:35:12.031343 (+ 900us) negotiation.cc:107] Waiting for socket to connect
0812 02:35:12.031390 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0812 02:35:12.032274 (+ 884us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:35:12.032923 (+ 649us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:35:12.032930 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:35:12.033345 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:35:12.033867 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:35:12.033881 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:35:12.034779 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:35:12.034782 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:35:12.035804 (+ 1022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:35:12.035815 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:35:12.036384 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:35:12.036388 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:35:12.036570 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:35:12.037266 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:35:12.037284 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:35:12.039293 (+ 2009us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:35:12.041493 (+ 2200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:35:12.041499 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:35:12.041515 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:35:12.041835 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:35:12.042263 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:35:12.042270 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:35:12.042275 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:35:12.042502 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:35:12.043118 (+ 616us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:35:12.043125 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:35:12.043415 (+ 290us) client_negotiation.cc:769] Sending connection context
0812 02:35:12.043597 (+ 182us) client_negotiation.cc:240] Negotiation successful
0812 02:35:12.043891 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":699,"thread_start_us":140,"threads_started":1}
I20260812 02:35:12.046277 32742 heartbeater.cc:347] Connected to a master server at 127.14.103.62:36667
I20260812 02:35:12.046556 32742 heartbeater.cc:464] Registering TS with master...
I20260812 02:35:12.047232 32742 heartbeater.cc:511] Master 127.14.103.62:36667 requested a full tablet report, sending...
I20260812 02:35:12.048401 32203 ts_manager.cc:194] Registered new tserver with Master: e8395e0332674f379f8c508b96e4e0c0 (127.14.103.3:33023)
I20260812 02:35:12.049022 32203 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.103.3@KRBTEST.COM'} at 127.14.103.3:52677
I20260812 02:35:12.053004 14748 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 02:35:12.063944 32246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:12.055377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37294 (local address 127.14.103.62:36667)
0812 02:35:12.055557 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0812 02:35:12.055561 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:35:12.055825 (+ 264us) server_negotiation.cc:407] Connection header received
0812 02:35:12.055926 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:35:12.055928 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:35:12.055974 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:35:12.056052 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:35:12.056759 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:35:12.057296 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:35:12.058181 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:35:12.058423 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:35:12.058521 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:35:12.060114 (+ 1593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:35:12.060117 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:35:12.060126 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:35:12.060161 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:35:12.061916 (+ 1755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:35:12.062462 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:35:12.062465 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:35:12.062466 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:35:12.062560 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:35:12.062916 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:35:12.062919 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:35:12.062921 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:35:12.063277 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:35:12.063402 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0812 02:35:12.063691 (+ 289us) server_negotiation.cc:299] Negotiation successful
0812 02:35:12.063809 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260812 02:35:12.067956 32203 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37294:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 02:35:12.071238 32203 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 02:35:12.087742 32757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:12.081746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40458 (local address 127.14.103.2:45799)
0812 02:35:12.082656 (+ 910us) server_negotiation.cc:206] Beginning negotiation
0812 02:35:12.082665 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 02:35:12.082682 (+ 17us) server_negotiation.cc:407] Connection header received
0812 02:35:12.082774 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:35:12.082806 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:35:12.083031 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:35:12.083198 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:35:12.083389 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:35:12.084533 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:35:12.086319 (+ 1786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:35:12.086924 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:35:12.087127 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:35:12.087297 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0812 02:35:12.087443 (+ 146us) server_negotiation.cc:299] Negotiation successful
0812 02:35:12.087565 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":824,"thread_start_us":107,"threads_started":1}
I20260812 02:35:12.087755 32753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:12.080805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.3:33023 (local address 127.0.0.1:34600)
0812 02:35:12.081275 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0812 02:35:12.081336 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0812 02:35:12.081471 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:35:12.082824 (+ 1353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:35:12.082830 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:35:12.082855 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:35:12.083081 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:35:12.083097 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:35:12.084720 (+ 1623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:35:12.084723 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:35:12.085999 (+ 1276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:35:12.086010 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:35:12.087195 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:35:12.087202 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:35:12.087337 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:35:12.087363 (+ 26us) client_negotiation.cc:769] Sending connection context
0812 02:35:12.087426 (+ 63us) client_negotiation.cc:240] Negotiation successful
0812 02:35:12.087578 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"mutex_wait_us":126,"thread_start_us":150,"threads_started":1}
I20260812 02:35:12.087774 32756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:12.081620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34600 (local address 127.14.103.3:33023)
0812 02:35:12.082267 (+ 647us) server_negotiation.cc:206] Beginning negotiation
0812 02:35:12.082280 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0812 02:35:12.082303 (+ 23us) server_negotiation.cc:407] Connection header received
0812 02:35:12.082384 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:35:12.082390 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:35:12.082613 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:35:12.082821 (+ 208us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:35:12.083234 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:35:12.084533 (+ 1299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:35:12.086130 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:35:12.087035 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:35:12.087337 (+ 302us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:35:12.087492 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0812 02:35:12.087547 (+ 55us) server_negotiation.cc:299] Negotiation successful
0812 02:35:12.087642 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":537,"thread_start_us":166,"threads_started":1}
I20260812 02:35:12.088163 32754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:12.080873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.1:37887 (local address 127.0.0.1:47364)
0812 02:35:12.081898 (+ 1025us) negotiation.cc:107] Waiting for socket to connect
0812 02:35:12.081912 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 02:35:12.082013 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:35:12.083023 (+ 1010us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:35:12.083026 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:35:12.083050 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:35:12.083166 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:35:12.083174 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:35:12.084354 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:35:12.084358 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:35:12.086144 (+ 1786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:35:12.086156 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:35:12.087278 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:35:12.087281 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:35:12.087864 (+ 583us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:35:12.087896 (+ 32us) client_negotiation.cc:769] Sending connection context
0812 02:35:12.087957 (+ 61us) client_negotiation.cc:240] Negotiation successful
0812 02:35:12.088027 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":962,"mutex_wait_us":232,"spinlock_wait_cycles":4864,"thread_start_us":129,"threads_started":1}
I20260812 02:35:12.088739 32758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:12.082056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47364 (local address 127.14.103.1:37887)
0812 02:35:12.082476 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0812 02:35:12.082482 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 02:35:12.082502 (+ 20us) server_negotiation.cc:407] Connection header received
0812 02:35:12.082565 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:35:12.082570 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:35:12.082804 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:35:12.083334 (+ 530us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:35:12.083481 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:35:12.084208 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:35:12.086266 (+ 2058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:35:12.087155 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:35:12.088267 (+ 1112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:35:12.088438 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0812 02:35:12.088496 (+ 58us) server_negotiation.cc:299] Negotiation successful
0812 02:35:12.088600 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":136,"threads_started":1}
I20260812 02:35:12.087743 32755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:12.081285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.2:45799 (local address 127.0.0.1:40458)
0812 02:35:12.082323 (+ 1038us) negotiation.cc:107] Waiting for socket to connect
0812 02:35:12.082337 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 02:35:12.082428 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:35:12.083185 (+ 757us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:35:12.083187 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:35:12.083205 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:35:12.083294 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:35:12.083300 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:35:12.084710 (+ 1410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:35:12.084714 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:35:12.086180 (+ 1466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:35:12.086190 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:35:12.087129 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:35:12.087133 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:35:12.087252 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:35:12.087290 (+ 38us) client_negotiation.cc:769] Sending connection context
0812 02:35:12.087411 (+ 121us) client_negotiation.cc:240] Negotiation successful
0812 02:35:12.087584 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":976,"mutex_wait_us":87,"thread_start_us":91,"threads_started":1}
I20260812 02:35:12.091830 32497 tablet_service.cc:1467] Processing CreateTablet for tablet 5ce157a2187d4e4b9d6aa8b15211b16d (DEFAULT_TABLE table=test-table [id=d1fd215f94a74d40b3b78bd1311f923f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:35:12.091701 32345 tablet_service.cc:1467] Processing CreateTablet for tablet 5ce157a2187d4e4b9d6aa8b15211b16d (DEFAULT_TABLE table=test-table [id=d1fd215f94a74d40b3b78bd1311f923f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:35:12.091701 32663 tablet_service.cc:1467] Processing CreateTablet for tablet 5ce157a2187d4e4b9d6aa8b15211b16d (DEFAULT_TABLE table=test-table [id=d1fd215f94a74d40b3b78bd1311f923f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:35:12.092944 32497 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ce157a2187d4e4b9d6aa8b15211b16d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:35:12.092945 32345 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ce157a2187d4e4b9d6aa8b15211b16d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:35:12.092955 32663 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ce157a2187d4e4b9d6aa8b15211b16d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:35:12.100258 32759 tablet_bootstrap.cc:492] T 5ce157a2187d4e4b9d6aa8b15211b16d P 4c0b137c9b9044e49094207590ce977e: Bootstrap starting.
I20260812 02:35:12.101696 32761 tablet_bootstrap.cc:492] T 5ce157a2187d4e4b9d6aa8b15211b16d P e8395e0332674f379f8c508b96e4e0c0: Bootstrap starting.
I20260812 02:35:12.102061 32760 tablet_bootstrap.cc:492] T 5ce157a2187d4e4b9d6aa8b15211b16d P eb3490e69b2a4d72947384cae82a0d4e: Bootstrap starting.
I20260812 02:35:12.103896 32759 tablet_bootstrap.cc:654] T 5ce157a2187d4e4b9d6aa8b15211b16d P 4c0b137c9b9044e49094207590ce977e: Neither blocks nor log segments found. Creating new log.
I20260812 02:35:12.104219 32760 tablet_bootstrap.cc:654] T 5ce157a2187d4e4b9d6aa8b15211b16d P eb3490e69b2a4d72947384cae82a0d4e: Neither blocks nor log segments found. Creating new log.
I20260812 02:35:12.104912 32759 log.cc:824] T 5ce157a2187d4e4b9d6aa8b15211b16d P 4c0b137c9b9044e49094207590ce977e: Log is configured to *not* fsync() on all Append() calls
I20260812 02:35:12.105183 32761 tablet_bootstrap.cc:654] T 5ce157a2187d4e4b9d6aa8b15211b16d P e8395e0332674f379f8c508b96e4e0c0: Neither blocks nor log segments found. Creating new log.
I20260812 02:35:12.105237 32760 log.cc:824] T 5ce157a2187d4e4b9d6aa8b15211b16d P eb3490e69b2a4d72947384cae82a0d4e: Log is configured to *not* fsync() on all Append() calls
I20260812 02:35:12.106071 32761 log.cc:824] T 5ce157a2187d4e4b9d6aa8b15211b16d P e8395e0332674f379f8c508b96e4e0c0: Log is configured to *not* fsync() on all Append() calls
I20260812 02:35:12.107411 32760 tablet_bootstrap.cc:492] T 5ce157a2187d4e4b9d6aa8b15211b16d P eb3490e69b2a4d72947384cae82a0d4e: No bootstrap required, opened a new log
I20260812 02:35:12.107442 32759 tablet_bootstrap.cc:492] T 5ce157a2187d4e4b9d6aa8b15211b16d P 4c0b137c9b9044e49094207590ce977e: No bootstrap required, opened a new log
I20260812 02:35:12.107652 32760 ts_tablet_manager.cc:1397] T 5ce157a2187d4e4b9d6aa8b15211b16d P eb3490e69b2a4d72947384cae82a0d4e: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260812 02:35:12.107654 32759 ts_tablet_manager.cc:1397] T 5ce157a2187d4e4b9d6aa8b15211b16d P 4c0b137c9b9044e49094207590ce977e: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260812 02:35:12.108475 32761 tablet_bootstrap.cc:492] T 5ce157a2187d4e4b9d6aa8b15211b16d P e8395e0332674f379f8c508b96e4e0c0: No bootstrap required, opened a new log
I20260812 02:35:12.108682 32761 ts_tablet_manager.cc:1397] T 5ce157a2187d4e4b9d6aa8b15211b16d P e8395e0332674f379f8c508b96e4e0c0: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260812 02:35:12.111905 32760 raft_consensus.cc:348] T 5ce157a2187d4e4b9d6aa8b15211b16d P eb3490e69b2a4d72947384cae82a0d4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8395e0332674f379f8c508b96e4e0c0" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 33023 } } peers { permanent_uuid: "eb3490e69b2a4d72947384cae82a0d4e" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 37887 } } peers { permanent_uuid: "4c0b137c9b9044e49094207590ce977e" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 45799 } }
I20260812 02:35:12.112248 32760 raft_consensus.cc:374] T 5ce157a2187d4e4b9d6aa8b15211b16d P eb3490e69b2a4d72947384cae82a0d4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:35:12.112313 32760 raft_consensus.cc:729] T 5ce157a2187d4e4b9d6aa8b15211b16d P eb3490e69b2a4d72947384cae82a0d4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb3490e69b2a4d72947384cae82a0d4e, State: Initialized, Role: FOLLOWER
I20260812 02:35:12.112236 32759 raft_consensus.cc:348] T 5ce157a2187d4e4b9d6aa8b15211b16d P 4c0b137c9b9044e49094207590ce977e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8395e0332674f379f8c508b96e4e0c0" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 33023 } } peers { permanent_uuid: "eb3490e69b2a4d72947384cae82a0d4e" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 37887 } } peers { permanent_uuid: "4c0b137c9b9044e49094207590ce977e" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 45799 } }
I20260812 02:35:12.112561 32759 raft_consensus.cc:374] T 5ce157a2187d4e4b9d6aa8b15211b16d P 4c0b137c9b9044e49094207590ce977e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:35:12.112618 32759 raft_consensus.cc:729] T 5ce157a2187d4e4b9d6aa8b15211b16d P 4c0b137c9b9044e49094207590ce977e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c0b137c9b9044e49094207590ce977e, State: Initialized, Role: FOLLOWER
I20260812 02:35:12.112869 32760 consensus_queue.cc:260] T 5ce157a2187d4e4b9d6aa8b15211b16d P eb3490e69b2a4d72947384cae82a0d4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8395e0332674f379f8c508b96e4e0c0" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 33023 } } peers { permanent_uuid: "eb3490e69b2a4d72947384cae82a0d4e" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 37887 } } peers { permanent_uuid: "4c0b137c9b9044e49094207590ce977e" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 45799 } }
I20260812 02:35:12.113185 32759 consensus_queue.cc:260] T 5ce157a2187d4e4b9d6aa8b15211b16d P 4c0b137c9b9044e49094207590ce977e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8395e0332674f379f8c508b96e4e0c0" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 33023 } } peers { permanent_uuid: "eb3490e69b2a4d72947384cae82a0d4e" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 37887 } } peers { permanent_uuid: "4c0b137c9b9044e49094207590ce977e" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 45799 } }
I20260812 02:35:12.113305 32761 raft_consensus.cc:348] T 5ce157a2187d4e4b9d6aa8b15211b16d P e8395e0332674f379f8c508b96e4e0c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8395e0332674f379f8c508b96e4e0c0" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 33023 } } peers { permanent_uuid: "eb3490e69b2a4d72947384cae82a0d4e" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 37887 } } peers { permanent_uuid: "4c0b137c9b9044e49094207590ce977e" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 45799 } }
I20260812 02:35:12.113624 32761 raft_consensus.cc:374] T 5ce157a2187d4e4b9d6aa8b15211b16d P e8395e0332674f379f8c508b96e4e0c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:35:12.113694 32761 raft_consensus.cc:729] T 5ce157a2187d4e4b9d6aa8b15211b16d P e8395e0332674f379f8c508b96e4e0c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8395e0332674f379f8c508b96e4e0c0, State: Initialized, Role: FOLLOWER
I20260812 02:35:12.113843 32418 heartbeater.cc:503] Master 127.14.103.62:36667 was elected leader, sending a full tablet report...
I20260812 02:35:12.114132 32760 ts_tablet_manager.cc:1428] T 5ce157a2187d4e4b9d6aa8b15211b16d P eb3490e69b2a4d72947384cae82a0d4e: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260812 02:35:12.114148 32761 consensus_queue.cc:260] T 5ce157a2187d4e4b9d6aa8b15211b16d P e8395e0332674f379f8c508b96e4e0c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8395e0332674f379f8c508b96e4e0c0" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 33023 } } peers { permanent_uuid: "eb3490e69b2a4d72947384cae82a0d4e" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 37887 } } peers { permanent_uuid: "4c0b137c9b9044e49094207590ce977e" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 45799 } }
I20260812 02:35:12.114252 32759 ts_tablet_manager.cc:1428] T 5ce157a2187d4e4b9d6aa8b15211b16d P 4c0b137c9b9044e49094207590ce977e: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260812 02:35:12.114594 32585 heartbeater.cc:503] Master 127.14.103.62:36667 was elected leader, sending a full tablet report...
I20260812 02:35:12.115033 32761 ts_tablet_manager.cc:1428] T 5ce157a2187d4e4b9d6aa8b15211b16d P e8395e0332674f379f8c508b96e4e0c0: Time spent starting tablet: real 0.006s user 0.004s sys 0.004s
I20260812 02:35:12.115356 32742 heartbeater.cc:503] Master 127.14.103.62:36667 was elected leader, sending a full tablet report...
I20260812 02:35:12.207072 32766 raft_consensus.cc:482] T 5ce157a2187d4e4b9d6aa8b15211b16d P eb3490e69b2a4d72947384cae82a0d4e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 02:35:12.207330 32766 raft_consensus.cc:504] T 5ce157a2187d4e4b9d6aa8b15211b16d P eb3490e69b2a4d72947384cae82a0d4e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8395e0332674f379f8c508b96e4e0c0" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 33023 } } peers { permanent_uuid: "eb3490e69b2a4d72947384cae82a0d4e" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 37887 } } peers { permanent_uuid: "4c0b137c9b9044e49094207590ce977e" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 45799 } }
I20260812 02:35:12.208576 32766 leader_election.cc:288] T 5ce157a2187d4e4b9d6aa8b15211b16d P eb3490e69b2a4d72947384cae82a0d4e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e8395e0332674f379f8c508b96e4e0c0 (127.14.103.3:33023), 4c0b137c9b9044e49094207590ce977e (127.14.103.2:45799)
I20260812 02:35:12.212983 32757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:12.209171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.1:34831 (local address 127.14.103.2:45799)
0812 02:35:12.209325 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0812 02:35:12.209329 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:35:12.209645 (+ 316us) server_negotiation.cc:407] Connection header received
0812 02:35:12.209811 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:35:12.209814 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:35:12.209876 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:35:12.209977 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:35:12.210242 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:35:12.210738 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:35:12.211961 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:35:12.212377 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:35:12.212517 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:35:12.212575 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0812 02:35:12.212763 (+ 188us) server_negotiation.cc:299] Negotiation successful
0812 02:35:12.212838 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260812 02:35:12.213006 303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:12.208950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.2:45799 (local address 127.14.103.1:34831)
0812 02:35:12.209524 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0812 02:35:12.209537 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 02:35:12.209701 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:35:12.209978 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:35:12.209981 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:35:12.209998 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:35:12.210125 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:35:12.210131 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:35:12.210863 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:35:12.210866 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:35:12.211821 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:35:12.211831 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:35:12.212517 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:35:12.212521 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:35:12.212621 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:35:12.212653 (+ 32us) client_negotiation.cc:769] Sending connection context
0812 02:35:12.212745 (+ 92us) client_negotiation.cc:240] Negotiation successful
0812 02:35:12.212824 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":499,"mutex_wait_us":132,"thread_start_us":112,"threads_started":1}
I20260812 02:35:12.213215 32756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:12.209189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.1:38111 (local address 127.14.103.3:33023)
0812 02:35:12.209382 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0812 02:35:12.209387 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:35:12.209514 (+ 127us) server_negotiation.cc:407] Connection header received
0812 02:35:12.209691 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:35:12.209694 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:35:12.209746 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:35:12.209833 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:35:12.210146 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:35:12.210884 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:35:12.212197 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:35:12.212646 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:35:12.212786 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:35:12.212835 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0812 02:35:12.213028 (+ 193us) server_negotiation.cc:299] Negotiation successful
0812 02:35:12.213104 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111}
I20260812 02:35:12.213232 302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:12.208930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.3:33023 (local address 127.14.103.1:38111)
0812 02:35:12.209393 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0812 02:35:12.209409 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 02:35:12.209574 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:35:12.209848 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:35:12.209852 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:35:12.209868 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:35:12.210024 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:35:12.210030 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:35:12.211245 (+ 1215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:35:12.211252 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:35:12.212048 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:35:12.212058 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:35:12.212780 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:35:12.212783 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:35:12.212886 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:35:12.212917 (+ 31us) client_negotiation.cc:769] Sending connection context
0812 02:35:12.213010 (+ 93us) client_negotiation.cc:240] Negotiation successful
0812 02:35:12.213085 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":159,"threads_started":1}
I20260812 02:35:12.214197 32522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ce157a2187d4e4b9d6aa8b15211b16d" candidate_uuid: "eb3490e69b2a4d72947384cae82a0d4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c0b137c9b9044e49094207590ce977e" is_pre_election: true
I20260812 02:35:12.214195 32685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ce157a2187d4e4b9d6aa8b15211b16d" candidate_uuid: "eb3490e69b2a4d72947384cae82a0d4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8395e0332674f379f8c508b96e4e0c0" is_pre_election: true
I20260812 02:35:12.214566 32522 raft_consensus.cc:2454] T 5ce157a2187d4e4b9d6aa8b15211b16d P 4c0b137c9b9044e49094207590ce977e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb3490e69b2a4d72947384cae82a0d4e in term 0.
I20260812 02:35:12.214566 32685 raft_consensus.cc:2454] T 5ce157a2187d4e4b9d6aa8b15211b16d P e8395e0332674f379f8c508b96e4e0c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb3490e69b2a4d72947384cae82a0d4e in term 0.
I20260812 02:35:12.215242 32293 leader_election.cc:302] T 5ce157a2187d4e4b9d6aa8b15211b16d P eb3490e69b2a4d72947384cae82a0d4e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 4c0b137c9b9044e49094207590ce977e, e8395e0332674f379f8c508b96e4e0c0, eb3490e69b2a4d72947384cae82a0d4e; no voters:
I20260812 02:35:12.215605 32766 raft_consensus.cc:2781] T 5ce157a2187d4e4b9d6aa8b15211b16d P eb3490e69b2a4d72947384cae82a0d4e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 02:35:12.215729 32766 raft_consensus.cc:482] T 5ce157a2187d4e4b9d6aa8b15211b16d P eb3490e69b2a4d72947384cae82a0d4e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 02:35:12.215821 32766 raft_consensus.cc:3037] T 5ce157a2187d4e4b9d6aa8b15211b16d P eb3490e69b2a4d72947384cae82a0d4e [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:35:12.217069 32766 raft_consensus.cc:504] T 5ce157a2187d4e4b9d6aa8b15211b16d P eb3490e69b2a4d72947384cae82a0d4e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8395e0332674f379f8c508b96e4e0c0" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 33023 } } peers { permanent_uuid: "eb3490e69b2a4d72947384cae82a0d4e" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 37887 } } peers { permanent_uuid: "4c0b137c9b9044e49094207590ce977e" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 45799 } }
I20260812 02:35:12.217522 32766 leader_election.cc:288] T 5ce157a2187d4e4b9d6aa8b15211b16d P eb3490e69b2a4d72947384cae82a0d4e [CANDIDATE]: Term 1 election: Requested vote from peers e8395e0332674f379f8c508b96e4e0c0 (127.14.103.3:33023), 4c0b137c9b9044e49094207590ce977e (127.14.103.2:45799)
I20260812 02:35:12.218109 32522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ce157a2187d4e4b9d6aa8b15211b16d" candidate_uuid: "eb3490e69b2a4d72947384cae82a0d4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c0b137c9b9044e49094207590ce977e"
I20260812 02:35:12.218107 32685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ce157a2187d4e4b9d6aa8b15211b16d" candidate_uuid: "eb3490e69b2a4d72947384cae82a0d4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8395e0332674f379f8c508b96e4e0c0"
I20260812 02:35:12.218268 32522 raft_consensus.cc:3037] T 5ce157a2187d4e4b9d6aa8b15211b16d P 4c0b137c9b9044e49094207590ce977e [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:35:12.218268 32685 raft_consensus.cc:3037] T 5ce157a2187d4e4b9d6aa8b15211b16d P e8395e0332674f379f8c508b96e4e0c0 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:35:12.219826 32685 raft_consensus.cc:2454] T 5ce157a2187d4e4b9d6aa8b15211b16d P e8395e0332674f379f8c508b96e4e0c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb3490e69b2a4d72947384cae82a0d4e in term 1.
I20260812 02:35:12.219826 32522 raft_consensus.cc:2454] T 5ce157a2187d4e4b9d6aa8b15211b16d P 4c0b137c9b9044e49094207590ce977e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb3490e69b2a4d72947384cae82a0d4e in term 1.
I20260812 02:35:12.220309 32293 leader_election.cc:302] T 5ce157a2187d4e4b9d6aa8b15211b16d P eb3490e69b2a4d72947384cae82a0d4e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e8395e0332674f379f8c508b96e4e0c0, eb3490e69b2a4d72947384cae82a0d4e; no voters:
I20260812 02:35:12.220587 32766 raft_consensus.cc:2781] T 5ce157a2187d4e4b9d6aa8b15211b16d P eb3490e69b2a4d72947384cae82a0d4e [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:35:12.220916 32766 raft_consensus.cc:686] T 5ce157a2187d4e4b9d6aa8b15211b16d P eb3490e69b2a4d72947384cae82a0d4e [term 1 LEADER]: Becoming Leader. State: Replica: eb3490e69b2a4d72947384cae82a0d4e, State: Running, Role: LEADER
I20260812 02:35:12.221346 32766 consensus_queue.cc:237] T 5ce157a2187d4e4b9d6aa8b15211b16d P eb3490e69b2a4d72947384cae82a0d4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8395e0332674f379f8c508b96e4e0c0" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 33023 } } peers { permanent_uuid: "eb3490e69b2a4d72947384cae82a0d4e" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 37887 } } peers { permanent_uuid: "4c0b137c9b9044e49094207590ce977e" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 45799 } }
I20260812 02:35:12.225190 32202 catalog_manager.cc:5495] T 5ce157a2187d4e4b9d6aa8b15211b16d P eb3490e69b2a4d72947384cae82a0d4e reported cstate change: term changed from 0 to 1, leader changed from <none> to eb3490e69b2a4d72947384cae82a0d4e (127.14.103.1). New cstate: current_term: 1 leader_uuid: "eb3490e69b2a4d72947384cae82a0d4e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8395e0332674f379f8c508b96e4e0c0" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 33023 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eb3490e69b2a4d72947384cae82a0d4e" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 37887 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4c0b137c9b9044e49094207590ce977e" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 45799 } health_report { overall_health: UNKNOWN } } }
I20260812 02:35:12.244946 32758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:12.240970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47374 (local address 127.14.103.1:37887)
0812 02:35:12.241198 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0812 02:35:12.241202 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:35:12.241217 (+ 15us) server_negotiation.cc:407] Connection header received
0812 02:35:12.241304 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:35:12.241306 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:35:12.241361 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:35:12.241491 (+ 130us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 02:35:12.241777 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:35:12.242293 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:35:12.243274 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:35:12.243526 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:35:12.243616 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:35:12.244026 (+ 410us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 02:35:12.244514 (+ 488us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 02:35:12.244613 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0812 02:35:12.244746 (+ 133us) server_negotiation.cc:299] Negotiation successful
0812 02:35:12.244809 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127}
I20260812 02:35:12.251874 32522 raft_consensus.cc:1261] T 5ce157a2187d4e4b9d6aa8b15211b16d P 4c0b137c9b9044e49094207590ce977e [term 1 FOLLOWER]: Refusing update from remote peer eb3490e69b2a4d72947384cae82a0d4e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:35:12.251869 32685 raft_consensus.cc:1261] T 5ce157a2187d4e4b9d6aa8b15211b16d P e8395e0332674f379f8c508b96e4e0c0 [term 1 FOLLOWER]: Refusing update from remote peer eb3490e69b2a4d72947384cae82a0d4e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:35:12.252859 304 consensus_queue.cc:1035] T 5ce157a2187d4e4b9d6aa8b15211b16d P eb3490e69b2a4d72947384cae82a0d4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8395e0332674f379f8c508b96e4e0c0" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 33023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:35:12.253208 32766 consensus_queue.cc:1035] T 5ce157a2187d4e4b9d6aa8b15211b16d P eb3490e69b2a4d72947384cae82a0d4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c0b137c9b9044e49094207590ce977e" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 45799 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:35:12.261463 310 mvcc.cc:205] Tried to move back new op lower bound from 7317512651774849024 to 7317512651661905920. Current Snapshot: MvccSnapshot[applied={T|T < 7317512651774849024}]
I20260812 02:35:12.261852 308 mvcc.cc:205] Tried to move back new op lower bound from 7317512651774849024 to 7317512651661905920. Current Snapshot: MvccSnapshot[applied={T|T < 7317512651774849024}]
I20260812 02:35:12.266621 309 mvcc.cc:205] Tried to move back new op lower bound from 7317512651774849024 to 7317512651661905920. Current Snapshot: MvccSnapshot[applied={T|T < 7317512651774849024 or (T in {7317512651774849024})}]
I20260812 02:35:12.271667 32203 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37294:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 02:35:12.271853 32203 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37294:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 02:35:12.275345 32203 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1806e8fbbfe747018262028788ff8fcf: Sending DeleteTablet for 3 replicas of tablet 5ce157a2187d4e4b9d6aa8b15211b16d
I20260812 02:35:12.276482 32663 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ce157a2187d4e4b9d6aa8b15211b16d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:35:12 UTC) from {username='kudu', principal='kudu/127.14.103.62@KRBTEST.COM'} at 127.0.0.1:34600
I20260812 02:35:12.276705 32497 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ce157a2187d4e4b9d6aa8b15211b16d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:35:12 UTC) from {username='kudu', principal='kudu/127.14.103.62@KRBTEST.COM'} at 127.0.0.1:40458
I20260812 02:35:12.277261 318 tablet_replica.cc:330] T 5ce157a2187d4e4b9d6aa8b15211b16d P 4c0b137c9b9044e49094207590ce977e: stopping tablet replica
I20260812 02:35:12.277406 14748 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbep4M4/build/debug/bin/kudu with pid 32273
I20260812 02:35:12.277371 32345 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ce157a2187d4e4b9d6aa8b15211b16d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:35:12 UTC) from {username='kudu', principal='kudu/127.14.103.62@KRBTEST.COM'} at 127.0.0.1:47364
I20260812 02:35:12.283756 317 tablet_replica.cc:330] T 5ce157a2187d4e4b9d6aa8b15211b16d P e8395e0332674f379f8c508b96e4e0c0: stopping tablet replica
I20260812 02:35:12.284190 317 raft_consensus.cc:2229] T 5ce157a2187d4e4b9d6aa8b15211b16d P e8395e0332674f379f8c508b96e4e0c0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 02:35:12.284562 317 raft_consensus.cc:2258] T 5ce157a2187d4e4b9d6aa8b15211b16d P e8395e0332674f379f8c508b96e4e0c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:35:12.286700 318 raft_consensus.cc:2229] T 5ce157a2187d4e4b9d6aa8b15211b16d P 4c0b137c9b9044e49094207590ce977e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 02:35:12.287017 317 ts_tablet_manager.cc:1905] T 5ce157a2187d4e4b9d6aa8b15211b16d P e8395e0332674f379f8c508b96e4e0c0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:35:12.287155 318 raft_consensus.cc:2258] T 5ce157a2187d4e4b9d6aa8b15211b16d P 4c0b137c9b9044e49094207590ce977e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:35:12.289013 318 ts_tablet_manager.cc:1905] T 5ce157a2187d4e4b9d6aa8b15211b16d P 4c0b137c9b9044e49094207590ce977e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:35:12.290112 317 ts_tablet_manager.cc:1918] T 5ce157a2187d4e4b9d6aa8b15211b16d P e8395e0332674f379f8c508b96e4e0c0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 02:35:12.290223 317 log.cc:1196] T 5ce157a2187d4e4b9d6aa8b15211b16d P e8395e0332674f379f8c508b96e4e0c0: Deleting WAL directory at /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/minicluster-data/ts-2/wal/wals/5ce157a2187d4e4b9d6aa8b15211b16d
I20260812 02:35:12.290580 317 ts_tablet_manager.cc:1939] T 5ce157a2187d4e4b9d6aa8b15211b16d P e8395e0332674f379f8c508b96e4e0c0: Deleting consensus metadata
I20260812 02:35:12.291970 32185 catalog_manager.cc:4841] TS e8395e0332674f379f8c508b96e4e0c0 (127.14.103.3:33023): tablet 5ce157a2187d4e4b9d6aa8b15211b16d (table test-table [id=d1fd215f94a74d40b3b78bd1311f923f]) successfully deleted
I20260812 02:35:12.292642 318 ts_tablet_manager.cc:1918] T 5ce157a2187d4e4b9d6aa8b15211b16d P 4c0b137c9b9044e49094207590ce977e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 02:35:12.292768 318 log.cc:1196] T 5ce157a2187d4e4b9d6aa8b15211b16d P 4c0b137c9b9044e49094207590ce977e: Deleting WAL directory at /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/minicluster-data/ts-1/wal/wals/5ce157a2187d4e4b9d6aa8b15211b16d
I20260812 02:35:12.293483 318 ts_tablet_manager.cc:1939] T 5ce157a2187d4e4b9d6aa8b15211b16d P 4c0b137c9b9044e49094207590ce977e: Deleting consensus metadata
I20260812 02:35:12.295424 32187 catalog_manager.cc:4841] TS 4c0b137c9b9044e49094207590ce977e (127.14.103.2:45799): tablet 5ce157a2187d4e4b9d6aa8b15211b16d (table test-table [id=d1fd215f94a74d40b3b78bd1311f923f]) successfully deleted
W20260812 02:35:12.295795 32184 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.103.1:37887 (error 108)
I20260812 02:35:12.296692 14748 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbep4M4/build/debug/bin/kudu with pid 32423
I20260812 02:35:12.296963 32755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:12.296587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.1:37887 (local address 127.0.0.1:47378)
0812 02:35:12.296757 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0812 02:35:12.296850 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.1:37887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260812 02:35:12.304677 32184 catalog_manager.cc:4584] TS eb3490e69b2a4d72947384cae82a0d4e (127.14.103.1:37887): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5ce157a2187d4e4b9d6aa8b15211b16d: Network error: Client connection negotiation failed: client connection to 127.14.103.1:37887: connect: Connection refused (error 111)
I20260812 02:35:12.309792 14748 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbep4M4/build/debug/bin/kudu with pid 32592
I20260812 02:35:12.320679 14748 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbep4M4/build/debug/bin/kudu with pid 32166
I20260812 02:35:32.332461 14748 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbep4M4/build/debug/bin/kudu
/tmp/dist-test-taskbep4M4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.103.62:36667
--webserver_interface=127.14.103.62
--webserver_port=34833
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.62.keytab
--principal=kudu/127.14.103.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.103.20:45573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.103.62:36667
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:35:32.463088 320 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:35:32.463476 320 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:35:32.463593 320 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:35:32.468415 320 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260812 02:35:32.468529 320 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260812 02:35:32.468562 320 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260812 02:35:32.468590 320 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:35:32.468642 320 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:35:32.468693 320 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:35:32.468739 320 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:35:32.468788 320 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 02:35:32.475514 320 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.103.20:45573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.14.103.62:36667
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.62.keytab
--principal=kudu/127.14.103.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.103.62:36667
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.103.62
--webserver_port=34833
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.320
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:35:32.476787 320 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:35:32.477934 320 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:35:32.493692 328 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:35:32.493705 327 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:35:32.493748 320 server_base.cc:1034] running on GCE node
W20260812 02:35:32.493981 332 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:35:32.494810 320 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:35:32.496184 320 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:35:32.497402 320 hybrid_clock.cc:648] HybridClock initialized: now 1786502132497344 us; error 55 us; skew 500 ppm
Aug 12 02:35:32 dist-test-slave-2w3w krb5kdc[32148](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502132, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:35:32.500612 320 init.cc:377] Logged in from keytab as kudu/127.14.103.62@KRBTEST.COM (short username kudu)
I20260812 02:35:32.501873 320 webserver.cc:460] Webserver started at http://127.14.103.62:34833/ using document root <none> and password file <none>
I20260812 02:35:32.502519 320 fs_manager.cc:359] Metadata directory not provided
I20260812 02:35:32.502645 320 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:35:32.507167 320 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260812 02:35:32.509941 342 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:35:32.511693 320 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 02:35:32.511893 320 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/minicluster-data/master-0/data,/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/minicluster-data/master-0/wal
uuid: "1806e8fbbfe747018262028788ff8fcf"
format_stamp: "Formatted at 2026-08-12 02:35:11 on dist-test-slave-2w3w"
server_key: "31908b9dfc9c08e00ad8c34bfd89f6ae"
server_key_iv: "6c2b36b4aacae94d61c9d53340224cd2"
server_key_version: "encryptionkey@0"
I20260812 02:35:32.512621 320 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:35:32.554986 320 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:35:32.558369 320 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:35:32.558606 320 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:35:32.566385 320 rpc_server.cc:307] RPC server started. Bound to: 127.14.103.62:36667
I20260812 02:35:32.566691 404 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.103.62:36667 every 8 connection(s)
I20260812 02:35:32.567675 320 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/minicluster-data/master-0/data/info.pb
I20260812 02:35:32.573069 14748 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbep4M4/build/debug/bin/kudu as pid 320
I20260812 02:35:32.573091 405 sys_catalog.cc:263] Verifying existing consensus state
I20260812 02:35:32.573616 14748 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbep4M4/build/debug/bin/kudu
/tmp/dist-test-taskbep4M4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.103.1:37887
--local_ip_for_outbound_sockets=127.14.103.1
--tserver_master_addrs=127.14.103.62:36667
--webserver_port=35679
--webserver_interface=127.14.103.1
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.1.keytab
--principal=kudu/127.14.103.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.103.20:45573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 02:35:32.575624 405 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1806e8fbbfe747018262028788ff8fcf: Bootstrap starting.
I20260812 02:35:32.594535 405 log.cc:824] T 00000000000000000000000000000000 P 1806e8fbbfe747018262028788ff8fcf: Log is configured to *not* fsync() on all Append() calls
I20260812 02:35:32.604013 405 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1806e8fbbfe747018262028788ff8fcf: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 02:35:32.604497 405 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1806e8fbbfe747018262028788ff8fcf: Bootstrap complete.
I20260812 02:35:32.609009 405 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1806e8fbbfe747018262028788ff8fcf [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1806e8fbbfe747018262028788ff8fcf" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 36667 } }
I20260812 02:35:32.609614 405 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1806e8fbbfe747018262028788ff8fcf [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1806e8fbbfe747018262028788ff8fcf, State: Initialized, Role: FOLLOWER
I20260812 02:35:32.610148 405 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1806e8fbbfe747018262028788ff8fcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1806e8fbbfe747018262028788ff8fcf" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 36667 } }
I20260812 02:35:32.610303 405 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1806e8fbbfe747018262028788ff8fcf [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 02:35:32.610416 405 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1806e8fbbfe747018262028788ff8fcf [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 02:35:32.610544 405 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1806e8fbbfe747018262028788ff8fcf [term 1 FOLLOWER]: Advancing to term 2
I20260812 02:35:32.612792 405 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1806e8fbbfe747018262028788ff8fcf [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1806e8fbbfe747018262028788ff8fcf" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 36667 } }
I20260812 02:35:32.613174 405 leader_election.cc:302] T 00000000000000000000000000000000 P 1806e8fbbfe747018262028788ff8fcf [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1806e8fbbfe747018262028788ff8fcf; no voters:
I20260812 02:35:32.613554 405 leader_election.cc:288] T 00000000000000000000000000000000 P 1806e8fbbfe747018262028788ff8fcf [CANDIDATE]: Term 2 election: Requested vote from peers
I20260812 02:35:32.613654 410 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1806e8fbbfe747018262028788ff8fcf [term 2 FOLLOWER]: Leader election won for term 2
I20260812 02:35:32.613875 410 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1806e8fbbfe747018262028788ff8fcf [term 2 LEADER]: Becoming Leader. State: Replica: 1806e8fbbfe747018262028788ff8fcf, State: Running, Role: LEADER
I20260812 02:35:32.614248 410 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1806e8fbbfe747018262028788ff8fcf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1806e8fbbfe747018262028788ff8fcf" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 36667 } }
I20260812 02:35:32.614939 405 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1806e8fbbfe747018262028788ff8fcf [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:35:32.615916 412 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1806e8fbbfe747018262028788ff8fcf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1806e8fbbfe747018262028788ff8fcf. Latest consensus state: current_term: 2 leader_uuid: "1806e8fbbfe747018262028788ff8fcf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1806e8fbbfe747018262028788ff8fcf" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 36667 } } }
I20260812 02:35:32.615967 411 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1806e8fbbfe747018262028788ff8fcf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "1806e8fbbfe747018262028788ff8fcf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1806e8fbbfe747018262028788ff8fcf" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 36667 } } }
I20260812 02:35:32.616052 412 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1806e8fbbfe747018262028788ff8fcf [sys.catalog]: This master's current role is: LEADER
I20260812 02:35:32.616091 411 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1806e8fbbfe747018262028788ff8fcf [sys.catalog]: This master's current role is: LEADER
I20260812 02:35:32.620738 428 catalog_manager.cc:1259] Loaded cluster ID: 4bc06eec6ed3497e94bf64b1bc9c3590
I20260812 02:35:32.620831 428 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 1806e8fbbfe747018262028788ff8fcf: loading cluster ID for follower catalog manager: success
I20260812 02:35:32.623596 428 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 1806e8fbbfe747018262028788ff8fcf: acquiring CA information for follower catalog manager: success
I20260812 02:35:32.625403 428 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 1806e8fbbfe747018262028788ff8fcf: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 02:35:32.625582 422 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 02:35:32.628368 422 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 02:35:32.628897 422 catalog_manager.cc:1259] Loaded cluster ID: 4bc06eec6ed3497e94bf64b1bc9c3590
I20260812 02:35:32.628960 422 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 02:35:32.629701 422 catalog_manager.cc:1504] Loading token signing keys...
I20260812 02:35:32.630225 422 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 1806e8fbbfe747018262028788ff8fcf: Loaded TSK: 0
I20260812 02:35:32.644142 422 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1806e8fbbfe747018262028788ff8fcf: Generated new TSK 1
I20260812 02:35:32.644492 422 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260812 02:35:32.712795 407 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:35:32.713099 407 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:35:32.713146 407 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:35:32.717926 407 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:35:32.718019 407 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:35:32.718086 407 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.103.1
I20260812 02:35:32.724408 407 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.103.20:45573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.1.keytab
--principal=kudu/127.14.103.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.103.1:37887
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.103.1
--webserver_port=35679
--tserver_master_addrs=127.14.103.62:36667
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.407
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.103.1
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:35:32.725899 407 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:35:32.727219 407 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:35:32.738724 438 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:35:32.738863 439 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:35:32.739020 441 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:35:32.739629 407 server_base.cc:1034] running on GCE node
I20260812 02:35:32.740152 407 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:35:32.740963 407 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:35:32.742185 407 hybrid_clock.cc:648] HybridClock initialized: now 1786502132742195 us; error 73 us; skew 500 ppm
Aug 12 02:35:32 dist-test-slave-2w3w krb5kdc[32148](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502132, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:35:32.745855 407 init.cc:377] Logged in from keytab as kudu/127.14.103.1@KRBTEST.COM (short username kudu)
I20260812 02:35:32.747076 407 webserver.cc:460] Webserver started at http://127.14.103.1:35679/ using document root <none> and password file <none>
I20260812 02:35:32.747794 407 fs_manager.cc:359] Metadata directory not provided
I20260812 02:35:32.747922 407 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:35:32.752630 407 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20260812 02:35:32.755688 449 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:35:32.760644 407 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.003s sys 0.000s
I20260812 02:35:32.760848 407 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/minicluster-data/ts-0/wal
uuid: "eb3490e69b2a4d72947384cae82a0d4e"
format_stamp: "Formatted at 2026-08-12 02:35:11 on dist-test-slave-2w3w"
server_key: "c676407f995241b3a413e8c7aabc506c"
server_key_iv: "e3fd61796ec1f02c47b288f367d1e9f9"
server_key_version: "encryptionkey@0"
I20260812 02:35:32.761907 407 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:35:32.798359 407 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:35:32.802129 407 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:35:32.802307 407 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:35:32.802878 407 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:35:32.804107 459 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260812 02:35:32.808930 407 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260812 02:35:32.809078 407 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260812 02:35:32.809219 407 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260812 02:35:32.811604 407 ts_tablet_manager.cc:610] Registered 1 tablets
I20260812 02:35:32.811697 407 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.002s sys 0.000s
I20260812 02:35:32.812028 459 tablet_bootstrap.cc:492] T 5ce157a2187d4e4b9d6aa8b15211b16d P eb3490e69b2a4d72947384cae82a0d4e: Bootstrap starting.
I20260812 02:35:32.824091 407 rpc_server.cc:307] RPC server started. Bound to: 127.14.103.1:37887
I20260812 02:35:32.824190 585 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.103.1:37887 every 8 connection(s)
I20260812 02:35:32.825122 407 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/minicluster-data/ts-0/data/info.pb
I20260812 02:35:32.831727 459 log.cc:824] T 5ce157a2187d4e4b9d6aa8b15211b16d P eb3490e69b2a4d72947384cae82a0d4e: Log is configured to *not* fsync() on all Append() calls
I20260812 02:35:32.833890 14748 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbep4M4/build/debug/bin/kudu as pid 407
I20260812 02:35:32.834432 14748 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbep4M4/build/debug/bin/kudu
/tmp/dist-test-taskbep4M4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.103.2:45799
--local_ip_for_outbound_sockets=127.14.103.2
--tserver_master_addrs=127.14.103.62:36667
--webserver_port=44487
--webserver_interface=127.14.103.2
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.2.keytab
--principal=kudu/127.14.103.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.103.20:45573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 02:35:32 dist-test-slave-2w3w krb5kdc[32148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502132, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.1@KRBTEST.COM for kudu/127.14.103.62@KRBTEST.COM
I20260812 02:35:32.842355 459 tablet_bootstrap.cc:492] T 5ce157a2187d4e4b9d6aa8b15211b16d P eb3490e69b2a4d72947384cae82a0d4e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 02:35:32.842922 459 tablet_bootstrap.cc:492] T 5ce157a2187d4e4b9d6aa8b15211b16d P eb3490e69b2a4d72947384cae82a0d4e: Bootstrap complete.
I20260812 02:35:32.844399 459 ts_tablet_manager.cc:1397] T 5ce157a2187d4e4b9d6aa8b15211b16d P eb3490e69b2a4d72947384cae82a0d4e: Time spent bootstrapping tablet: real 0.033s user 0.011s sys 0.020s
I20260812 02:35:32.846361 588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:32.827672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.1:33335 (local address 127.14.103.62:36667)
0812 02:35:32.828091 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0812 02:35:32.828111 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0812 02:35:32.828471 (+ 360us) server_negotiation.cc:407] Connection header received
0812 02:35:32.829691 (+ 1220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:35:32.829702 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:35:32.830017 (+ 315us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:35:32.830367 (+ 350us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:35:32.831955 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:35:32.832985 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:35:32.834788 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:35:32.835569 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:35:32.835759 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:35:32.838703 (+ 2944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:35:32.838710 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:35:32.838727 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:35:32.838767 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:35:32.841928 (+ 3161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:35:32.842841 (+ 913us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:35:32.842851 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:35:32.842860 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:35:32.843090 (+ 230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:35:32.843582 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:35:32.843588 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:35:32.843593 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:35:32.844133 (+ 540us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:35:32.844275 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0812 02:35:32.845066 (+ 791us) server_negotiation.cc:299] Negotiation successful
0812 02:35:32.845410 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256,"thread_start_us":153,"threads_started":1}
I20260812 02:35:32.846465 589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:32.827684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:36667 (local address 127.14.103.1:33335)
0812 02:35:32.828289 (+ 605us) negotiation.cc:107] Waiting for socket to connect
0812 02:35:32.828337 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0812 02:35:32.829250 (+ 913us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:35:32.830413 (+ 1163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:35:32.830421 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:35:32.830898 (+ 477us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:35:32.831725 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:35:32.831747 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:35:32.833177 (+ 1430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:35:32.833180 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:35:32.834392 (+ 1212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:35:32.834406 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:35:32.835763 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:35:32.835770 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:35:32.836022 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:35:32.836764 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:35:32.836782 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:35:32.838410 (+ 1628us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:35:32.842128 (+ 3718us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:35:32.842138 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:35:32.842158 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:35:32.842671 (+ 513us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:35:32.843221 (+ 550us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:35:32.843227 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:35:32.843232 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:35:32.843456 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:35:32.844294 (+ 838us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:35:32.844302 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:35:32.844669 (+ 367us) client_negotiation.cc:769] Sending connection context
0812 02:35:32.844920 (+ 251us) client_negotiation.cc:240] Negotiation successful
0812 02:35:32.845373 (+ 453us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":141,"threads_started":1}
I20260812 02:35:32.847558 459 raft_consensus.cc:348] T 5ce157a2187d4e4b9d6aa8b15211b16d P eb3490e69b2a4d72947384cae82a0d4e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8395e0332674f379f8c508b96e4e0c0" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 33023 } } peers { permanent_uuid: "eb3490e69b2a4d72947384cae82a0d4e" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 37887 } } peers { permanent_uuid: "4c0b137c9b9044e49094207590ce977e" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 45799 } }
I20260812 02:35:32.848124 459 raft_consensus.cc:729] T 5ce157a2187d4e4b9d6aa8b15211b16d P eb3490e69b2a4d72947384cae82a0d4e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb3490e69b2a4d72947384cae82a0d4e, State: Initialized, Role: FOLLOWER
I20260812 02:35:32.848628 459 consensus_queue.cc:260] T 5ce157a2187d4e4b9d6aa8b15211b16d P eb3490e69b2a4d72947384cae82a0d4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8395e0332674f379f8c508b96e4e0c0" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 33023 } } peers { permanent_uuid: "eb3490e69b2a4d72947384cae82a0d4e" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 37887 } } peers { permanent_uuid: "4c0b137c9b9044e49094207590ce977e" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 45799 } }
I20260812 02:35:32.849759 586 heartbeater.cc:347] Connected to a master server at 127.14.103.62:36667
I20260812 02:35:32.849787 459 ts_tablet_manager.cc:1428] T 5ce157a2187d4e4b9d6aa8b15211b16d P eb3490e69b2a4d72947384cae82a0d4e: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20260812 02:35:32.850042 586 heartbeater.cc:464] Registering TS with master...
I20260812 02:35:32.850673 586 heartbeater.cc:511] Master 127.14.103.62:36667 requested a full tablet report, sending...
I20260812 02:35:32.853611 360 ts_manager.cc:194] Registered new tserver with Master: eb3490e69b2a4d72947384cae82a0d4e (127.14.103.1:37887)
I20260812 02:35:32.857453 360 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.103.1@KRBTEST.COM'} at 127.14.103.1:33335
I20260812 02:35:32.860047 586 heartbeater.cc:503] Master 127.14.103.62:36667 was elected leader, sending a full tablet report...
Aug 12 02:35:32 dist-test-slave-2w3w krb5kdc[32148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502132, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.62@KRBTEST.COM for kudu/127.14.103.1@KRBTEST.COM
I20260812 02:35:32.872087 594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:32.856737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.1:37887 (local address 127.0.0.1:57744)
0812 02:35:32.857618 (+ 881us) negotiation.cc:107] Waiting for socket to connect
0812 02:35:32.857663 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0812 02:35:32.857812 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:35:32.860561 (+ 2749us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:35:32.860571 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:35:32.861047 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:35:32.861193 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:35:32.861202 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:35:32.862909 (+ 1707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:35:32.862912 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:35:32.864216 (+ 1304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:35:32.864230 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:35:32.865141 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:35:32.865144 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:35:32.865242 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:35:32.865935 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:35:32.865949 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:35:32.867464 (+ 1515us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:35:32.869779 (+ 2315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:35:32.869792 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:35:32.869798 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:35:32.870191 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:35:32.870562 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:35:32.870566 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:35:32.870568 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:35:32.870673 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:35:32.871343 (+ 670us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:35:32.871350 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:35:32.871497 (+ 147us) client_negotiation.cc:769] Sending connection context
0812 02:35:32.871643 (+ 146us) client_negotiation.cc:240] Negotiation successful
0812 02:35:32.871886 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":706,"thread_start_us":147,"threads_started":1}
I20260812 02:35:32.872102 595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:32.857003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57744 (local address 127.14.103.1:37887)
0812 02:35:32.857437 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0812 02:35:32.857449 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 02:35:32.857780 (+ 331us) server_negotiation.cc:407] Connection header received
0812 02:35:32.857988 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:35:32.857997 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:35:32.858419 (+ 422us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:35:32.858569 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:35:32.861369 (+ 2800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:35:32.862689 (+ 1320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:35:32.864367 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:35:32.864897 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:35:32.865031 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:35:32.867641 (+ 2610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:35:32.867657 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:35:32.867670 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:35:32.867702 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:35:32.869613 (+ 1911us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:35:32.870358 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:35:32.870364 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:35:32.870365 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:35:32.870459 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:35:32.870814 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:35:32.870817 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:35:32.870818 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:35:32.871217 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:35:32.871337 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0812 02:35:32.871764 (+ 427us) server_negotiation.cc:299] Negotiation successful
0812 02:35:32.871935 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":101,"threads_started":1}
I20260812 02:35:32.873509 509 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ce157a2187d4e4b9d6aa8b15211b16d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:35:12 UTC) from {username='kudu', principal='kudu/127.14.103.62@KRBTEST.COM'} at 127.0.0.1:57744
I20260812 02:35:32.874471 596 tablet_replica.cc:330] stopping tablet replica
I20260812 02:35:32.874856 596 raft_consensus.cc:2229] T 5ce157a2187d4e4b9d6aa8b15211b16d P eb3490e69b2a4d72947384cae82a0d4e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 02:35:32.875162 596 raft_consensus.cc:2258] T 5ce157a2187d4e4b9d6aa8b15211b16d P eb3490e69b2a4d72947384cae82a0d4e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:35:32.876276 596 ts_tablet_manager.cc:1905] T 5ce157a2187d4e4b9d6aa8b15211b16d P eb3490e69b2a4d72947384cae82a0d4e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:35:32.880426 596 ts_tablet_manager.cc:1918] T 5ce157a2187d4e4b9d6aa8b15211b16d P eb3490e69b2a4d72947384cae82a0d4e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 02:35:32.880532 596 log.cc:1196] T 5ce157a2187d4e4b9d6aa8b15211b16d P eb3490e69b2a4d72947384cae82a0d4e: Deleting WAL directory at /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/minicluster-data/ts-0/wal/wals/5ce157a2187d4e4b9d6aa8b15211b16d
I20260812 02:35:32.880822 596 ts_tablet_manager.cc:1939] T 5ce157a2187d4e4b9d6aa8b15211b16d P eb3490e69b2a4d72947384cae82a0d4e: Deleting consensus metadata
I20260812 02:35:32.882655 343 catalog_manager.cc:4841] TS eb3490e69b2a4d72947384cae82a0d4e (127.14.103.1:37887): tablet 5ce157a2187d4e4b9d6aa8b15211b16d (table test-table [id=d1fd215f94a74d40b3b78bd1311f923f]) successfully deleted
W20260812 02:35:32.965971 592 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:35:32.966328 592 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:35:32.966427 592 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:35:32.971197 592 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:35:32.971323 592 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:35:32.971429 592 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.103.2
I20260812 02:35:32.977272 592 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.103.20:45573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.2.keytab
--principal=kudu/127.14.103.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.103.2:45799
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.103.2
--webserver_port=44487
--tserver_master_addrs=127.14.103.62:36667
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.592
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.103.2
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:35:32.978770 592 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:35:32.980083 592 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:35:32.992061 601 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:35:32.992120 604 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:35:32.992061 602 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:35:32.992724 592 server_base.cc:1034] running on GCE node
I20260812 02:35:32.993263 592 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:35:32.994037 592 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:35:32.995242 592 hybrid_clock.cc:648] HybridClock initialized: now 1786502132995207 us; error 55 us; skew 500 ppm
Aug 12 02:35:32 dist-test-slave-2w3w krb5kdc[32148](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502132, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:35:32.998366 592 init.cc:377] Logged in from keytab as kudu/127.14.103.2@KRBTEST.COM (short username kudu)
I20260812 02:35:32.999506 592 webserver.cc:460] Webserver started at http://127.14.103.2:44487/ using document root <none> and password file <none>
I20260812 02:35:33.000113 592 fs_manager.cc:359] Metadata directory not provided
I20260812 02:35:33.000236 592 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:35:33.004333 592 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260812 02:35:33.006673 612 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:35:33.008136 592 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 02:35:33.008250 592 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/minicluster-data/ts-1/wal
uuid: "4c0b137c9b9044e49094207590ce977e"
format_stamp: "Formatted at 2026-08-12 02:35:11 on dist-test-slave-2w3w"
server_key: "f4f69eb05b1f444046bfdcdd6e2e1ce8"
server_key_iv: "793d2d7b936db6cd2822f67e83dc527e"
server_key_version: "encryptionkey@0"
I20260812 02:35:33.008816 592 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:35:33.031684 592 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:35:33.035229 592 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:35:33.035440 592 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:35:33.036043 592 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:35:33.037149 592 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:35:33.037200 592 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:35:33.037240 592 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:35:33.037256 592 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:35:33.047693 592 rpc_server.cc:307] RPC server started. Bound to: 127.14.103.2:45799
I20260812 02:35:33.047760 749 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.103.2:45799 every 8 connection(s)
I20260812 02:35:33.049186 592 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/minicluster-data/ts-1/data/info.pb
I20260812 02:35:33.056695 14748 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbep4M4/build/debug/bin/kudu as pid 592
I20260812 02:35:33.057190 14748 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbep4M4/build/debug/bin/kudu
/tmp/dist-test-taskbep4M4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.103.3:33023
--local_ip_for_outbound_sockets=127.14.103.3
--tserver_master_addrs=127.14.103.62:36667
--webserver_port=44141
--webserver_interface=127.14.103.3
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.3.keytab
--principal=kudu/127.14.103.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.103.20:45573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 02:35:33 dist-test-slave-2w3w krb5kdc[32148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502132, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.2@KRBTEST.COM for kudu/127.14.103.62@KRBTEST.COM
I20260812 02:35:33.065879 588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:33.051424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.2:34863 (local address 127.14.103.62:36667)
0812 02:35:33.053083 (+ 1659us) server_negotiation.cc:206] Beginning negotiation
0812 02:35:33.053090 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 02:35:33.053112 (+ 22us) server_negotiation.cc:407] Connection header received
0812 02:35:33.054421 (+ 1309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:35:33.054425 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:35:33.054481 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:35:33.054618 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:35:33.055957 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:35:33.056529 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:35:33.057921 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:35:33.058172 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:35:33.058289 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:35:33.061422 (+ 3133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:35:33.061429 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:35:33.061435 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:35:33.061472 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:35:33.063425 (+ 1953us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:35:33.064014 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:35:33.064020 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:35:33.064024 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:35:33.064134 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:35:33.064502 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:35:33.064509 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:35:33.064512 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:35:33.064807 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:35:33.064931 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0812 02:35:33.065458 (+ 527us) server_negotiation.cc:299] Negotiation successful
0812 02:35:33.065636 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1559}
I20260812 02:35:33.066642 752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:33.051949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:36667 (local address 127.14.103.2:34863)
0812 02:35:33.052535 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0812 02:35:33.052598 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0812 02:35:33.054205 (+ 1607us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:35:33.054819 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:35:33.054832 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:35:33.055279 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:35:33.055787 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:35:33.055801 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:35:33.056664 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:35:33.056668 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:35:33.057779 (+ 1111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:35:33.057790 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:35:33.058334 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:35:33.058338 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:35:33.058521 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:35:33.059232 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:35:33.059249 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:35:33.061216 (+ 1967us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:35:33.063569 (+ 2353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:35:33.063575 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:35:33.063590 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:35:33.063887 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:35:33.064243 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:35:33.064245 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:35:33.064246 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:35:33.064397 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:35:33.064928 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:35:33.064934 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:35:33.065197 (+ 263us) client_negotiation.cc:769] Sending connection context
0812 02:35:33.065369 (+ 172us) client_negotiation.cc:240] Negotiation successful
0812 02:35:33.065658 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":156,"threads_started":1}
I20260812 02:35:33.067978 750 heartbeater.cc:347] Connected to a master server at 127.14.103.62:36667
I20260812 02:35:33.068241 750 heartbeater.cc:464] Registering TS with master...
I20260812 02:35:33.068765 750 heartbeater.cc:511] Master 127.14.103.62:36667 requested a full tablet report, sending...
I20260812 02:35:33.069911 360 ts_manager.cc:194] Registered new tserver with Master: 4c0b137c9b9044e49094207590ce977e (127.14.103.2:45799)
I20260812 02:35:33.070833 360 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.103.2@KRBTEST.COM'} at 127.14.103.2:34863
W20260812 02:35:33.178725 754 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:35:33.179124 754 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:35:33.179176 754 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:35:33.183264 754 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:35:33.183333 754 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:35:33.183396 754 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.103.3
I20260812 02:35:33.188905 754 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.103.20:45573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.3.keytab
--principal=kudu/127.14.103.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.103.3:33023
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.103.3
--webserver_port=44141
--tserver_master_addrs=127.14.103.62:36667
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.754
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.103.3
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:35:33.190003 754 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:35:33.191061 754 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:35:33.201330 763 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:35:33.201347 761 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:35:33.201329 760 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:35:33.201716 754 server_base.cc:1034] running on GCE node
I20260812 02:35:33.202251 754 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:35:33.203013 754 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:35:33.204242 754 hybrid_clock.cc:648] HybridClock initialized: now 1786502133204223 us; error 38 us; skew 500 ppm
Aug 12 02:35:33 dist-test-slave-2w3w krb5kdc[32148](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502133, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:35:33.207258 754 init.cc:377] Logged in from keytab as kudu/127.14.103.3@KRBTEST.COM (short username kudu)
I20260812 02:35:33.208249 754 webserver.cc:460] Webserver started at http://127.14.103.3:44141/ using document root <none> and password file <none>
I20260812 02:35:33.208801 754 fs_manager.cc:359] Metadata directory not provided
I20260812 02:35:33.208884 754 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:35:33.213016 754 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260812 02:35:33.215528 771 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:35:33.216976 754 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 02:35:33.217120 754 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/minicluster-data/ts-2/wal
uuid: "e8395e0332674f379f8c508b96e4e0c0"
format_stamp: "Formatted at 2026-08-12 02:35:11 on dist-test-slave-2w3w"
server_key: "985211a092ad46521774423ece2bee0a"
server_key_iv: "a128ee492a442ed5171334ed7cbc8269"
server_key_version: "encryptionkey@0"
I20260812 02:35:33.217741 754 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:35:33.232502 754 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:35:33.235630 754 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:35:33.235801 754 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:35:33.236361 754 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:35:33.237166 754 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:35:33.237212 754 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:35:33.237254 754 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:35:33.237278 754 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:35:33.247437 754 rpc_server.cc:307] RPC server started. Bound to: 127.14.103.3:33023
I20260812 02:35:33.247490 902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.103.3:33023 every 8 connection(s)
I20260812 02:35:33.248520 754 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/minicluster-data/ts-2/data/info.pb
I20260812 02:35:33.256081 14748 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbep4M4/build/debug/bin/kudu as pid 754
Aug 12 02:35:33 dist-test-slave-2w3w krb5kdc[32148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502133, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.3@KRBTEST.COM for kudu/127.14.103.62@KRBTEST.COM
I20260812 02:35:33.264614 588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:33.250585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.3:58485 (local address 127.14.103.62:36667)
0812 02:35:33.250748 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0812 02:35:33.250753 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:35:33.251801 (+ 1048us) server_negotiation.cc:407] Connection header received
0812 02:35:33.253280 (+ 1479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:35:33.253283 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:35:33.253415 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:35:33.253506 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:35:33.255215 (+ 1709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:35:33.255777 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:35:33.257449 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:35:33.257817 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:35:33.257961 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:35:33.260646 (+ 2685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:35:33.260650 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:35:33.260653 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:35:33.260685 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:35:33.262318 (+ 1633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:35:33.262859 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:35:33.262867 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:35:33.262869 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:35:33.263028 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:35:33.263391 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:35:33.263394 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:35:33.263396 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:35:33.263588 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:35:33.263702 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0812 02:35:33.264275 (+ 573us) server_negotiation.cc:299] Negotiation successful
0812 02:35:33.264386 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260812 02:35:33.265620 905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:33.251083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:36667 (local address 127.14.103.3:58485)
0812 02:35:33.251581 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0812 02:35:33.251637 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0812 02:35:33.252974 (+ 1337us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:35:33.253724 (+ 750us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:35:33.253732 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:35:33.254186 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:35:33.254876 (+ 690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:35:33.254901 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:35:33.255970 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:35:33.255977 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:35:33.257285 (+ 1308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:35:33.257298 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:35:33.258018 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:35:33.258022 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:35:33.258193 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:35:33.258863 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:35:33.258881 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:35:33.260474 (+ 1593us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:35:33.262444 (+ 1970us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:35:33.262449 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:35:33.262464 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:35:33.262738 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:35:33.263130 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:35:33.263133 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:35:33.263135 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:35:33.263286 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:35:33.263700 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:35:33.263707 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:35:33.263984 (+ 277us) client_negotiation.cc:769] Sending connection context
0812 02:35:33.264210 (+ 226us) client_negotiation.cc:240] Negotiation successful
0812 02:35:33.264572 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":151,"threads_started":1}
I20260812 02:35:33.265937 907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:33.256990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44384 (local address 127.14.103.62:36667)
0812 02:35:33.257303 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0812 02:35:33.257307 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:35:33.257489 (+ 182us) server_negotiation.cc:407] Connection header received
0812 02:35:33.257654 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:35:33.257656 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:35:33.257704 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:35:33.257792 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:35:33.258732 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:35:33.259381 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:35:33.260435 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:35:33.260775 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:35:33.260917 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:35:33.262388 (+ 1471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:35:33.262394 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:35:33.262397 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:35:33.262428 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:35:33.263993 (+ 1565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:35:33.264465 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:35:33.264468 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:35:33.264470 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:35:33.264552 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:35:33.264986 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:35:33.264989 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:35:33.264990 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:35:33.265149 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:35:33.265300 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0812 02:35:33.265520 (+ 220us) server_negotiation.cc:299] Negotiation successful
0812 02:35:33.265618 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":219,"thread_start_us":102,"threads_started":1}
I20260812 02:35:33.267047 903 heartbeater.cc:347] Connected to a master server at 127.14.103.62:36667
I20260812 02:35:33.267361 903 heartbeater.cc:464] Registering TS with master...
I20260812 02:35:33.267983 903 heartbeater.cc:511] Master 127.14.103.62:36667 requested a full tablet report, sending...
I20260812 02:35:33.269124 359 ts_manager.cc:194] Registered new tserver with Master: e8395e0332674f379f8c508b96e4e0c0 (127.14.103.3:33023)
I20260812 02:35:33.269852 359 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.103.3@KRBTEST.COM'} at 127.14.103.3:58485
I20260812 02:35:33.280326 14748 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 02:35:33.284716 907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:33.281016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44390 (local address 127.14.103.62:36667)
0812 02:35:33.281176 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0812 02:35:33.281180 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:35:33.281354 (+ 174us) server_negotiation.cc:407] Connection header received
0812 02:35:33.281505 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:35:33.281508 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:35:33.281616 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:35:33.281710 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 02:35:33.281969 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:35:33.282493 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:35:33.283446 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:35:33.283702 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:35:33.283850 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:35:33.284176 (+ 326us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 02:35:33.284330 (+ 154us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0812 02:35:33.284581 (+ 251us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:44390: token expired
Metrics: {"server-negotiator.queue_time_us":63}
W20260812 02:35:33.284850 907 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:44390: token expired
W20260812 02:35:33.285297 32748 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.14.103.62:36667)
I20260812 02:35:33.293529 907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:33.285756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44392 (local address 127.14.103.62:36667)
0812 02:35:33.285910 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0812 02:35:33.285915 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:35:33.285929 (+ 14us) server_negotiation.cc:407] Connection header received
0812 02:35:33.286008 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:35:33.286011 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:35:33.286057 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:35:33.286142 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:35:33.286784 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:35:33.287381 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:35:33.288181 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:35:33.288399 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:35:33.288502 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:35:33.289991 (+ 1489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:35:33.289995 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:35:33.289997 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:35:33.290022 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:35:33.291665 (+ 1643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:35:33.292186 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:35:33.292189 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:35:33.292191 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:35:33.292286 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:35:33.292572 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:35:33.292575 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:35:33.292577 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:35:33.292776 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:35:33.292961 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0812 02:35:33.293207 (+ 246us) server_negotiation.cc:299] Negotiation successful
0812 02:35:33.293342 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260812 02:35:33.296474 359 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44392:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 02:35:33.299914 359 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 02:35:33.308496 509 tablet_service.cc:1467] Processing CreateTablet for tablet c057e89d50e8481fa551ffa7a5d6d166 (DEFAULT_TABLE table=test-table [id=1d480991dfb248adbe6c7f8620c250ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:35:33.309208 509 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c057e89d50e8481fa551ffa7a5d6d166. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:35:33.312253 459 tablet_bootstrap.cc:492] T c057e89d50e8481fa551ffa7a5d6d166 P eb3490e69b2a4d72947384cae82a0d4e: Bootstrap starting.
I20260812 02:35:33.313189 594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:33.307091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.2:45799 (local address 127.0.0.1:59484)
0812 02:35:33.307208 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0812 02:35:33.307224 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 02:35:33.307313 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:35:33.308284 (+ 971us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:35:33.308287 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:35:33.308300 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:35:33.308455 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:35:33.308461 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:35:33.310156 (+ 1695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:35:33.310159 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:35:33.311577 (+ 1418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:35:33.311590 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:35:33.312733 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:35:33.312737 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:35:33.312873 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:35:33.312931 (+ 58us) client_negotiation.cc:769] Sending connection context
0812 02:35:33.312986 (+ 55us) client_negotiation.cc:240] Negotiation successful
0812 02:35:33.313077 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33,"mutex_wait_us":25}
I20260812 02:35:33.313328 459 tablet_bootstrap.cc:654] T c057e89d50e8481fa551ffa7a5d6d166 P eb3490e69b2a4d72947384cae82a0d4e: Neither blocks nor log segments found. Creating new log.
I20260812 02:35:33.313812 916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:33.307412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.3:33023 (local address 127.0.0.1:57960)
0812 02:35:33.308176 (+ 764us) negotiation.cc:107] Waiting for socket to connect
0812 02:35:33.308188 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 02:35:33.308298 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:35:33.308997 (+ 699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:35:33.308999 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:35:33.309015 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:35:33.309087 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:35:33.309092 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:35:33.310350 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:35:33.310357 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:35:33.311610 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:35:33.311621 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:35:33.313161 (+ 1540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:35:33.313167 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:35:33.313297 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:35:33.313327 (+ 30us) client_negotiation.cc:769] Sending connection context
0812 02:35:33.313406 (+ 79us) client_negotiation.cc:240] Negotiation successful
0812 02:35:33.313564 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":700,"mutex_wait_us":20,"thread_start_us":142,"threads_started":1}
I20260812 02:35:33.314218 915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:33.307272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59484 (local address 127.14.103.2:45799)
0812 02:35:33.307581 (+ 309us) server_negotiation.cc:206] Beginning negotiation
0812 02:35:33.307595 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0812 02:35:33.307621 (+ 26us) server_negotiation.cc:407] Connection header received
0812 02:35:33.307856 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:35:33.307864 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:35:33.308110 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:35:33.308286 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:35:33.308553 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:35:33.309798 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:35:33.311725 (+ 1927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:35:33.312582 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:35:33.313716 (+ 1134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:35:33.313911 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0812 02:35:33.313979 (+ 68us) server_negotiation.cc:299] Negotiation successful
0812 02:35:33.314079 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":214,"thread_start_us":123,"threads_started":1}
I20260812 02:35:33.314308 917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:33.307712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57960 (local address 127.14.103.3:33023)
0812 02:35:33.308060 (+ 348us) server_negotiation.cc:206] Beginning negotiation
0812 02:35:33.308073 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0812 02:35:33.308395 (+ 322us) server_negotiation.cc:407] Connection header received
0812 02:35:33.308476 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:35:33.308484 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:35:33.308663 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:35:33.308822 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:35:33.309205 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:35:33.310244 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:35:33.311741 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:35:33.312821 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:35:33.312939 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:35:33.313091 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0812 02:35:33.314018 (+ 927us) server_negotiation.cc:299] Negotiation successful
0812 02:35:33.314131 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":252,"thread_start_us":155,"threads_started":1}
I20260812 02:35:33.315369 459 tablet_bootstrap.cc:492] T c057e89d50e8481fa551ffa7a5d6d166 P eb3490e69b2a4d72947384cae82a0d4e: No bootstrap required, opened a new log
I20260812 02:35:33.315474 459 ts_tablet_manager.cc:1397] T c057e89d50e8481fa551ffa7a5d6d166 P eb3490e69b2a4d72947384cae82a0d4e: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260812 02:35:33.316017 459 raft_consensus.cc:348] T c057e89d50e8481fa551ffa7a5d6d166 P eb3490e69b2a4d72947384cae82a0d4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c0b137c9b9044e49094207590ce977e" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 45799 } } peers { permanent_uuid: "e8395e0332674f379f8c508b96e4e0c0" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 33023 } } peers { permanent_uuid: "eb3490e69b2a4d72947384cae82a0d4e" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 37887 } }
I20260812 02:35:33.316149 459 raft_consensus.cc:374] T c057e89d50e8481fa551ffa7a5d6d166 P eb3490e69b2a4d72947384cae82a0d4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:35:33.316188 459 raft_consensus.cc:729] T c057e89d50e8481fa551ffa7a5d6d166 P eb3490e69b2a4d72947384cae82a0d4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb3490e69b2a4d72947384cae82a0d4e, State: Initialized, Role: FOLLOWER
I20260812 02:35:33.316588 459 consensus_queue.cc:260] T c057e89d50e8481fa551ffa7a5d6d166 P eb3490e69b2a4d72947384cae82a0d4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c0b137c9b9044e49094207590ce977e" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 45799 } } peers { permanent_uuid: "e8395e0332674f379f8c508b96e4e0c0" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 33023 } } peers { permanent_uuid: "eb3490e69b2a4d72947384cae82a0d4e" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 37887 } }
I20260812 02:35:33.316867 459 ts_tablet_manager.cc:1428] T c057e89d50e8481fa551ffa7a5d6d166 P eb3490e69b2a4d72947384cae82a0d4e: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260812 02:35:33.317517 831 tablet_service.cc:1467] Processing CreateTablet for tablet c057e89d50e8481fa551ffa7a5d6d166 (DEFAULT_TABLE table=test-table [id=1d480991dfb248adbe6c7f8620c250ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:35:33.317503 672 tablet_service.cc:1467] Processing CreateTablet for tablet c057e89d50e8481fa551ffa7a5d6d166 (DEFAULT_TABLE table=test-table [id=1d480991dfb248adbe6c7f8620c250ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:35:33.318511 831 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c057e89d50e8481fa551ffa7a5d6d166. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:35:33.318511 672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c057e89d50e8481fa551ffa7a5d6d166. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:35:33.326315 920 tablet_bootstrap.cc:492] T c057e89d50e8481fa551ffa7a5d6d166 P e8395e0332674f379f8c508b96e4e0c0: Bootstrap starting.
W20260812 02:35:33.326625 587 tablet.cc:2369] T c057e89d50e8481fa551ffa7a5d6d166 P eb3490e69b2a4d72947384cae82a0d4e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 02:35:33.327468 922 tablet_bootstrap.cc:492] T c057e89d50e8481fa551ffa7a5d6d166 P 4c0b137c9b9044e49094207590ce977e: Bootstrap starting.
I20260812 02:35:33.328655 920 tablet_bootstrap.cc:654] T c057e89d50e8481fa551ffa7a5d6d166 P e8395e0332674f379f8c508b96e4e0c0: Neither blocks nor log segments found. Creating new log.
I20260812 02:35:33.329511 922 tablet_bootstrap.cc:654] T c057e89d50e8481fa551ffa7a5d6d166 P 4c0b137c9b9044e49094207590ce977e: Neither blocks nor log segments found. Creating new log.
I20260812 02:35:33.329820 920 log.cc:824] T c057e89d50e8481fa551ffa7a5d6d166 P e8395e0332674f379f8c508b96e4e0c0: Log is configured to *not* fsync() on all Append() calls
I20260812 02:35:33.330205 922 log.cc:824] T c057e89d50e8481fa551ffa7a5d6d166 P 4c0b137c9b9044e49094207590ce977e: Log is configured to *not* fsync() on all Append() calls
I20260812 02:35:33.331974 920 tablet_bootstrap.cc:492] T c057e89d50e8481fa551ffa7a5d6d166 P e8395e0332674f379f8c508b96e4e0c0: No bootstrap required, opened a new log
I20260812 02:35:33.332091 922 tablet_bootstrap.cc:492] T c057e89d50e8481fa551ffa7a5d6d166 P 4c0b137c9b9044e49094207590ce977e: No bootstrap required, opened a new log
I20260812 02:35:33.332186 920 ts_tablet_manager.cc:1397] T c057e89d50e8481fa551ffa7a5d6d166 P e8395e0332674f379f8c508b96e4e0c0: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260812 02:35:33.332309 922 ts_tablet_manager.cc:1397] T c057e89d50e8481fa551ffa7a5d6d166 P 4c0b137c9b9044e49094207590ce977e: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260812 02:35:33.335323 920 raft_consensus.cc:348] T c057e89d50e8481fa551ffa7a5d6d166 P e8395e0332674f379f8c508b96e4e0c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c0b137c9b9044e49094207590ce977e" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 45799 } } peers { permanent_uuid: "e8395e0332674f379f8c508b96e4e0c0" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 33023 } } peers { permanent_uuid: "eb3490e69b2a4d72947384cae82a0d4e" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 37887 } }
I20260812 02:35:33.335323 922 raft_consensus.cc:348] T c057e89d50e8481fa551ffa7a5d6d166 P 4c0b137c9b9044e49094207590ce977e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c0b137c9b9044e49094207590ce977e" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 45799 } } peers { permanent_uuid: "e8395e0332674f379f8c508b96e4e0c0" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 33023 } } peers { permanent_uuid: "eb3490e69b2a4d72947384cae82a0d4e" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 37887 } }
I20260812 02:35:33.335615 922 raft_consensus.cc:374] T c057e89d50e8481fa551ffa7a5d6d166 P 4c0b137c9b9044e49094207590ce977e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:35:33.335623 920 raft_consensus.cc:374] T c057e89d50e8481fa551ffa7a5d6d166 P e8395e0332674f379f8c508b96e4e0c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:35:33.335711 922 raft_consensus.cc:729] T c057e89d50e8481fa551ffa7a5d6d166 P 4c0b137c9b9044e49094207590ce977e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c0b137c9b9044e49094207590ce977e, State: Initialized, Role: FOLLOWER
I20260812 02:35:33.335727 920 raft_consensus.cc:729] T c057e89d50e8481fa551ffa7a5d6d166 P e8395e0332674f379f8c508b96e4e0c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8395e0332674f379f8c508b96e4e0c0, State: Initialized, Role: FOLLOWER
I20260812 02:35:33.336210 922 consensus_queue.cc:260] T c057e89d50e8481fa551ffa7a5d6d166 P 4c0b137c9b9044e49094207590ce977e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c0b137c9b9044e49094207590ce977e" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 45799 } } peers { permanent_uuid: "e8395e0332674f379f8c508b96e4e0c0" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 33023 } } peers { permanent_uuid: "eb3490e69b2a4d72947384cae82a0d4e" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 37887 } }
I20260812 02:35:33.336211 920 consensus_queue.cc:260] T c057e89d50e8481fa551ffa7a5d6d166 P e8395e0332674f379f8c508b96e4e0c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c0b137c9b9044e49094207590ce977e" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 45799 } } peers { permanent_uuid: "e8395e0332674f379f8c508b96e4e0c0" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 33023 } } peers { permanent_uuid: "eb3490e69b2a4d72947384cae82a0d4e" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 37887 } }
I20260812 02:35:33.336982 903 heartbeater.cc:503] Master 127.14.103.62:36667 was elected leader, sending a full tablet report...
I20260812 02:35:33.337268 920 ts_tablet_manager.cc:1428] T c057e89d50e8481fa551ffa7a5d6d166 P e8395e0332674f379f8c508b96e4e0c0: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260812 02:35:33.337284 922 ts_tablet_manager.cc:1428] T c057e89d50e8481fa551ffa7a5d6d166 P 4c0b137c9b9044e49094207590ce977e: Time spent starting tablet: real 0.005s user 0.001s sys 0.004s
I20260812 02:35:33.338074 750 heartbeater.cc:503] Master 127.14.103.62:36667 was elected leader, sending a full tablet report...
W20260812 02:35:33.499691 904 tablet.cc:2369] T c057e89d50e8481fa551ffa7a5d6d166 P e8395e0332674f379f8c508b96e4e0c0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 02:35:33.551234 751 tablet.cc:2369] T c057e89d50e8481fa551ffa7a5d6d166 P 4c0b137c9b9044e49094207590ce977e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 02:35:33.692467 928 raft_consensus.cc:482] T c057e89d50e8481fa551ffa7a5d6d166 P 4c0b137c9b9044e49094207590ce977e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 02:35:33.692685 928 raft_consensus.cc:504] T c057e89d50e8481fa551ffa7a5d6d166 P 4c0b137c9b9044e49094207590ce977e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c0b137c9b9044e49094207590ce977e" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 45799 } } peers { permanent_uuid: "e8395e0332674f379f8c508b96e4e0c0" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 33023 } } peers { permanent_uuid: "eb3490e69b2a4d72947384cae82a0d4e" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 37887 } }
I20260812 02:35:33.693969 928 leader_election.cc:288] T c057e89d50e8481fa551ffa7a5d6d166 P 4c0b137c9b9044e49094207590ce977e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e8395e0332674f379f8c508b96e4e0c0 (127.14.103.3:33023), eb3490e69b2a4d72947384cae82a0d4e (127.14.103.1:37887)
I20260812 02:35:33.698364 917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:33.694280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.2:52793 (local address 127.14.103.3:33023)
0812 02:35:33.694485 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0812 02:35:33.694489 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:35:33.694602 (+ 113us) server_negotiation.cc:407] Connection header received
0812 02:35:33.694826 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:35:33.694833 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:35:33.694892 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:35:33.695022 (+ 130us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:35:33.695383 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:35:33.695920 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:35:33.697253 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:35:33.697675 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:35:33.697778 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:35:33.697862 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0812 02:35:33.698122 (+ 260us) server_negotiation.cc:299] Negotiation successful
0812 02:35:33.698211 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109}
I20260812 02:35:33.698375 930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:33.694282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.1:37887 (local address 127.14.103.2:59649)
0812 02:35:33.694628 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0812 02:35:33.694643 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 02:35:33.694800 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:35:33.695146 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:35:33.695148 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:35:33.695173 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:35:33.695341 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:35:33.695348 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:35:33.696267 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:35:33.696269 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:35:33.697104 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:35:33.697113 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:35:33.697888 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:35:33.697893 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:35:33.698014 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:35:33.698032 (+ 18us) client_negotiation.cc:769] Sending connection context
0812 02:35:33.698134 (+ 102us) client_negotiation.cc:240] Negotiation successful
0812 02:35:33.698232 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"mutex_wait_us":87,"thread_start_us":85,"threads_started":1}
I20260812 02:35:33.698369 931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:33.694516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.2:59649 (local address 127.14.103.1:37887)
0812 02:35:33.694802 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0812 02:35:33.694812 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 02:35:33.694825 (+ 13us) server_negotiation.cc:407] Connection header received
0812 02:35:33.694884 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:35:33.694886 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:35:33.694972 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:35:33.695122 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:35:33.695470 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:35:33.696041 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:35:33.697246 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:35:33.697716 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:35:33.697870 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:35:33.698049 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0812 02:35:33.698157 (+ 108us) server_negotiation.cc:299] Negotiation successful
0812 02:35:33.698237 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":89,"threads_started":1}
I20260812 02:35:33.698381 929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:33.694155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.3:33023 (local address 127.14.103.2:52793)
0812 02:35:33.694483 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0812 02:35:33.694501 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 02:35:33.694672 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:35:33.695014 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:35:33.695017 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:35:33.695034 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:35:33.695252 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:35:33.695259 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:35:33.696042 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:35:33.696046 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:35:33.697099 (+ 1053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:35:33.697110 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:35:33.697816 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:35:33.697823 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:35:33.697973 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:35:33.698000 (+ 27us) client_negotiation.cc:769] Sending connection context
0812 02:35:33.698081 (+ 81us) client_negotiation.cc:240] Negotiation successful
0812 02:35:33.698198 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"mutex_wait_us":73,"thread_start_us":91,"threads_started":1}
I20260812 02:35:33.699442 853 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c057e89d50e8481fa551ffa7a5d6d166" candidate_uuid: "4c0b137c9b9044e49094207590ce977e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8395e0332674f379f8c508b96e4e0c0" is_pre_election: true
I20260812 02:35:33.699437 530 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c057e89d50e8481fa551ffa7a5d6d166" candidate_uuid: "4c0b137c9b9044e49094207590ce977e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb3490e69b2a4d72947384cae82a0d4e" is_pre_election: true
I20260812 02:35:33.699791 853 raft_consensus.cc:2454] T c057e89d50e8481fa551ffa7a5d6d166 P e8395e0332674f379f8c508b96e4e0c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c0b137c9b9044e49094207590ce977e in term 0.
I20260812 02:35:33.699791 530 raft_consensus.cc:2454] T c057e89d50e8481fa551ffa7a5d6d166 P eb3490e69b2a4d72947384cae82a0d4e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c0b137c9b9044e49094207590ce977e in term 0.
I20260812 02:35:33.700371 613 leader_election.cc:302] T c057e89d50e8481fa551ffa7a5d6d166 P 4c0b137c9b9044e49094207590ce977e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c0b137c9b9044e49094207590ce977e, eb3490e69b2a4d72947384cae82a0d4e; no voters:
I20260812 02:35:33.700727 928 raft_consensus.cc:2781] T c057e89d50e8481fa551ffa7a5d6d166 P 4c0b137c9b9044e49094207590ce977e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 02:35:33.700829 928 raft_consensus.cc:482] T c057e89d50e8481fa551ffa7a5d6d166 P 4c0b137c9b9044e49094207590ce977e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 02:35:33.700902 928 raft_consensus.cc:3037] T c057e89d50e8481fa551ffa7a5d6d166 P 4c0b137c9b9044e49094207590ce977e [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:35:33.702045 928 raft_consensus.cc:504] T c057e89d50e8481fa551ffa7a5d6d166 P 4c0b137c9b9044e49094207590ce977e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c0b137c9b9044e49094207590ce977e" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 45799 } } peers { permanent_uuid: "e8395e0332674f379f8c508b96e4e0c0" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 33023 } } peers { permanent_uuid: "eb3490e69b2a4d72947384cae82a0d4e" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 37887 } }
I20260812 02:35:33.702500 928 leader_election.cc:288] T c057e89d50e8481fa551ffa7a5d6d166 P 4c0b137c9b9044e49094207590ce977e [CANDIDATE]: Term 1 election: Requested vote from peers e8395e0332674f379f8c508b96e4e0c0 (127.14.103.3:33023), eb3490e69b2a4d72947384cae82a0d4e (127.14.103.1:37887)
I20260812 02:35:33.703037 853 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c057e89d50e8481fa551ffa7a5d6d166" candidate_uuid: "4c0b137c9b9044e49094207590ce977e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8395e0332674f379f8c508b96e4e0c0"
I20260812 02:35:33.703051 530 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c057e89d50e8481fa551ffa7a5d6d166" candidate_uuid: "4c0b137c9b9044e49094207590ce977e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb3490e69b2a4d72947384cae82a0d4e"
I20260812 02:35:33.703217 530 raft_consensus.cc:3037] T c057e89d50e8481fa551ffa7a5d6d166 P eb3490e69b2a4d72947384cae82a0d4e [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:35:33.703217 853 raft_consensus.cc:3037] T c057e89d50e8481fa551ffa7a5d6d166 P e8395e0332674f379f8c508b96e4e0c0 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:35:33.704666 530 raft_consensus.cc:2454] T c057e89d50e8481fa551ffa7a5d6d166 P eb3490e69b2a4d72947384cae82a0d4e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c0b137c9b9044e49094207590ce977e in term 1.
I20260812 02:35:33.704666 853 raft_consensus.cc:2454] T c057e89d50e8481fa551ffa7a5d6d166 P e8395e0332674f379f8c508b96e4e0c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c0b137c9b9044e49094207590ce977e in term 1.
I20260812 02:35:33.705118 615 leader_election.cc:302] T c057e89d50e8481fa551ffa7a5d6d166 P 4c0b137c9b9044e49094207590ce977e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 4c0b137c9b9044e49094207590ce977e, e8395e0332674f379f8c508b96e4e0c0, eb3490e69b2a4d72947384cae82a0d4e; no voters:
I20260812 02:35:33.705348 928 raft_consensus.cc:2781] T c057e89d50e8481fa551ffa7a5d6d166 P 4c0b137c9b9044e49094207590ce977e [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:35:33.705628 928 raft_consensus.cc:686] T c057e89d50e8481fa551ffa7a5d6d166 P 4c0b137c9b9044e49094207590ce977e [term 1 LEADER]: Becoming Leader. State: Replica: 4c0b137c9b9044e49094207590ce977e, State: Running, Role: LEADER
I20260812 02:35:33.705998 928 consensus_queue.cc:237] T c057e89d50e8481fa551ffa7a5d6d166 P 4c0b137c9b9044e49094207590ce977e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c0b137c9b9044e49094207590ce977e" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 45799 } } peers { permanent_uuid: "e8395e0332674f379f8c508b96e4e0c0" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 33023 } } peers { permanent_uuid: "eb3490e69b2a4d72947384cae82a0d4e" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 37887 } }
I20260812 02:35:33.709659 360 catalog_manager.cc:5495] T c057e89d50e8481fa551ffa7a5d6d166 P 4c0b137c9b9044e49094207590ce977e reported cstate change: term changed from 0 to 1, leader changed from <none> to 4c0b137c9b9044e49094207590ce977e (127.14.103.2). New cstate: current_term: 1 leader_uuid: "4c0b137c9b9044e49094207590ce977e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c0b137c9b9044e49094207590ce977e" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 45799 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e8395e0332674f379f8c508b96e4e0c0" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 33023 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eb3490e69b2a4d72947384cae82a0d4e" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 37887 } health_report { overall_health: UNKNOWN } } }
I20260812 02:35:33.771970 915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:33.767978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59494 (local address 127.14.103.2:45799)
0812 02:35:33.768127 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0812 02:35:33.768132 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:35:33.768148 (+ 16us) server_negotiation.cc:407] Connection header received
0812 02:35:33.768366 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:35:33.768369 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:35:33.768431 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:35:33.768544 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 02:35:33.768858 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:35:33.769372 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:35:33.770290 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:35:33.770536 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:35:33.770633 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:35:33.771060 (+ 427us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 02:35:33.771519 (+ 459us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 02:35:33.771623 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0812 02:35:33.771734 (+ 111us) server_negotiation.cc:299] Negotiation successful
0812 02:35:33.771795 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260812 02:35:33.778465 530 raft_consensus.cc:1261] T c057e89d50e8481fa551ffa7a5d6d166 P eb3490e69b2a4d72947384cae82a0d4e [term 1 FOLLOWER]: Refusing update from remote peer 4c0b137c9b9044e49094207590ce977e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:35:33.778461 853 raft_consensus.cc:1261] T c057e89d50e8481fa551ffa7a5d6d166 P e8395e0332674f379f8c508b96e4e0c0 [term 1 FOLLOWER]: Refusing update from remote peer 4c0b137c9b9044e49094207590ce977e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:35:33.779273 928 consensus_queue.cc:1035] T c057e89d50e8481fa551ffa7a5d6d166 P 4c0b137c9b9044e49094207590ce977e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8395e0332674f379f8c508b96e4e0c0" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 33023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:35:33.779538 932 consensus_queue.cc:1035] T c057e89d50e8481fa551ffa7a5d6d166 P 4c0b137c9b9044e49094207590ce977e [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb3490e69b2a4d72947384cae82a0d4e" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 37887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:35:33.786918 935 mvcc.cc:205] Tried to move back new op lower bound from 7317512739948318720 to 7317512739662450688. Current Snapshot: MvccSnapshot[applied={T|T < 7317512739948318720}]
I20260812 02:35:33.799876 359 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44392:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 02:35:33.800114 359 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44392:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 02:35:33.802932 359 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1806e8fbbfe747018262028788ff8fcf: Sending DeleteTablet for 3 replicas of tablet c057e89d50e8481fa551ffa7a5d6d166
I20260812 02:35:33.803934 831 tablet_service.cc:1514] Processing DeleteTablet for tablet c057e89d50e8481fa551ffa7a5d6d166 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:35:33 UTC) from {username='kudu', principal='kudu/127.14.103.62@KRBTEST.COM'} at 127.0.0.1:57960
I20260812 02:35:33.804018 509 tablet_service.cc:1514] Processing DeleteTablet for tablet c057e89d50e8481fa551ffa7a5d6d166 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:35:33 UTC) from {username='kudu', principal='kudu/127.14.103.62@KRBTEST.COM'} at 127.0.0.1:57744
I20260812 02:35:33.803939 672 tablet_service.cc:1514] Processing DeleteTablet for tablet c057e89d50e8481fa551ffa7a5d6d166 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:35:33 UTC) from {username='kudu', principal='kudu/127.14.103.62@KRBTEST.COM'} at 127.0.0.1:59484
I20260812 02:35:33.804301 14748 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbep4M4/build/debug/bin/kudu with pid 407
I20260812 02:35:33.805819 948 tablet_replica.cc:330] T c057e89d50e8481fa551ffa7a5d6d166 P 4c0b137c9b9044e49094207590ce977e: stopping tablet replica
I20260812 02:35:33.806345 948 raft_consensus.cc:2229] T c057e89d50e8481fa551ffa7a5d6d166 P 4c0b137c9b9044e49094207590ce977e [term 1 LEADER]: Raft consensus shutting down.
I20260812 02:35:33.807035 948 raft_consensus.cc:2258] T c057e89d50e8481fa551ffa7a5d6d166 P 4c0b137c9b9044e49094207590ce977e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:35:33.807526 949 tablet_replica.cc:330] T c057e89d50e8481fa551ffa7a5d6d166 P e8395e0332674f379f8c508b96e4e0c0: stopping tablet replica
I20260812 02:35:33.808005 949 raft_consensus.cc:2229] T c057e89d50e8481fa551ffa7a5d6d166 P e8395e0332674f379f8c508b96e4e0c0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 02:35:33.808449 949 raft_consensus.cc:2258] T c057e89d50e8481fa551ffa7a5d6d166 P e8395e0332674f379f8c508b96e4e0c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:35:33.809715 948 ts_tablet_manager.cc:1905] T c057e89d50e8481fa551ffa7a5d6d166 P 4c0b137c9b9044e49094207590ce977e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:35:33.813982 948 ts_tablet_manager.cc:1918] T c057e89d50e8481fa551ffa7a5d6d166 P 4c0b137c9b9044e49094207590ce977e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 02:35:33.814206 948 log.cc:1196] T c057e89d50e8481fa551ffa7a5d6d166 P 4c0b137c9b9044e49094207590ce977e: Deleting WAL directory at /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/minicluster-data/ts-1/wal/wals/c057e89d50e8481fa551ffa7a5d6d166
I20260812 02:35:33.814633 948 ts_tablet_manager.cc:1939] T c057e89d50e8481fa551ffa7a5d6d166 P 4c0b137c9b9044e49094207590ce977e: Deleting consensus metadata
I20260812 02:35:33.816318 346 catalog_manager.cc:4841] TS 4c0b137c9b9044e49094207590ce977e (127.14.103.2:45799): tablet c057e89d50e8481fa551ffa7a5d6d166 (table test-table [id=1d480991dfb248adbe6c7f8620c250ec]) successfully deleted
W20260812 02:35:33.816980 343 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.103.1:37887 (error 108)
I20260812 02:35:33.817977 14748 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbep4M4/build/debug/bin/kudu with pid 592
I20260812 02:35:33.818349 953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:33.817805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.1:37887 (local address 127.0.0.1:57746)
0812 02:35:33.818107 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0812 02:35:33.818204 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.1:37887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":67,"threads_started":1}
W20260812 02:35:33.818501 343 catalog_manager.cc:4584] TS eb3490e69b2a4d72947384cae82a0d4e (127.14.103.1:37887): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c057e89d50e8481fa551ffa7a5d6d166: Network error: Client connection negotiation failed: client connection to 127.14.103.1:37887: connect: Connection refused (error 111)
I20260812 02:35:33.824070 949 ts_tablet_manager.cc:1905] T c057e89d50e8481fa551ffa7a5d6d166 P e8395e0332674f379f8c508b96e4e0c0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:35:33.826915 949 ts_tablet_manager.cc:1918] T c057e89d50e8481fa551ffa7a5d6d166 P e8395e0332674f379f8c508b96e4e0c0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 02:35:33.827114 949 log.cc:1196] T c057e89d50e8481fa551ffa7a5d6d166 P e8395e0332674f379f8c508b96e4e0c0: Deleting WAL directory at /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/minicluster-data/ts-2/wal/wals/c057e89d50e8481fa551ffa7a5d6d166
I20260812 02:35:33.827463 949 ts_tablet_manager.cc:1939] T c057e89d50e8481fa551ffa7a5d6d166 P e8395e0332674f379f8c508b96e4e0c0: Deleting consensus metadata
I20260812 02:35:33.829025 344 catalog_manager.cc:4841] TS e8395e0332674f379f8c508b96e4e0c0 (127.14.103.3:33023): tablet c057e89d50e8481fa551ffa7a5d6d166 (table test-table [id=1d480991dfb248adbe6c7f8620c250ec]) successfully deleted
I20260812 02:35:33.829281 14748 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbep4M4/build/debug/bin/kudu with pid 754
I20260812 02:35:33.840415 14748 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbep4M4/build/debug/bin/kudu with pid 320
I20260812 02:35:53.850383 14748 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbep4M4/build/debug/bin/kudu
/tmp/dist-test-taskbep4M4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.103.62:36667
--webserver_interface=127.14.103.62
--webserver_port=34833
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.62.keytab
--principal=kudu/127.14.103.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.103.20:45573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.103.62:36667
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:35:53.971900 954 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:35:53.972237 954 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:35:53.972318 954 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:35:53.977020 954 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260812 02:35:53.977092 954 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260812 02:35:53.977108 954 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260812 02:35:53.977123 954 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:35:53.977142 954 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:35:53.977196 954 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:35:53.977232 954 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:35:53.977253 954 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 02:35:53.982848 954 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.103.20:45573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.14.103.62:36667
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.62.keytab
--principal=kudu/127.14.103.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.103.62:36667
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.103.62
--webserver_port=34833
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.954
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:35:53.984422 954 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:35:53.985565 954 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:35:53.995630 960 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:35:53.995692 954 server_base.cc:1034] running on GCE node
W20260812 02:35:53.995630 959 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:35:53.995630 964 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:35:53.996696 954 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:35:53.998116 954 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:35:53.999315 954 hybrid_clock.cc:648] HybridClock initialized: now 1786502153999293 us; error 53 us; skew 500 ppm
Aug 12 02:35:53 dist-test-slave-2w3w krb5kdc[32148](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502153, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:35:54.002568 954 init.cc:377] Logged in from keytab as kudu/127.14.103.62@KRBTEST.COM (short username kudu)
I20260812 02:35:54.003628 954 webserver.cc:460] Webserver started at http://127.14.103.62:34833/ using document root <none> and password file <none>
I20260812 02:35:54.004197 954 fs_manager.cc:359] Metadata directory not provided
I20260812 02:35:54.004263 954 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:35:54.008965 954 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260812 02:35:54.011531 973 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:35:54.013116 954 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 02:35:54.013289 954 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/minicluster-data/master-0/data,/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/minicluster-data/master-0/wal
uuid: "1806e8fbbfe747018262028788ff8fcf"
format_stamp: "Formatted at 2026-08-12 02:35:11 on dist-test-slave-2w3w"
server_key: "31908b9dfc9c08e00ad8c34bfd89f6ae"
server_key_iv: "6c2b36b4aacae94d61c9d53340224cd2"
server_key_version: "encryptionkey@0"
I20260812 02:35:54.013917 954 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:35:54.027526 954 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:35:54.031241 954 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:35:54.031494 954 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:35:54.040405 954 rpc_server.cc:307] RPC server started. Bound to: 127.14.103.62:36667
I20260812 02:35:54.040766 1033 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.103.62:36667 every 8 connection(s)
I20260812 02:35:54.041666 954 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/minicluster-data/master-0/data/info.pb
I20260812 02:35:54.046459 1034 sys_catalog.cc:263] Verifying existing consensus state
I20260812 02:35:54.048207 1034 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1806e8fbbfe747018262028788ff8fcf: Bootstrap starting.
I20260812 02:35:54.049466 14748 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbep4M4/build/debug/bin/kudu as pid 954
I20260812 02:35:54.049991 14748 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbep4M4/build/debug/bin/kudu
/tmp/dist-test-taskbep4M4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.103.1:37887
--local_ip_for_outbound_sockets=127.14.103.1
--tserver_master_addrs=127.14.103.62:36667
--webserver_port=35679
--webserver_interface=127.14.103.1
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.1.keytab
--principal=kudu/127.14.103.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.103.20:45573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 02:35:54.067771 1034 log.cc:824] T 00000000000000000000000000000000 P 1806e8fbbfe747018262028788ff8fcf: Log is configured to *not* fsync() on all Append() calls
I20260812 02:35:54.079224 1034 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1806e8fbbfe747018262028788ff8fcf: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 02:35:54.079679 1034 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1806e8fbbfe747018262028788ff8fcf: Bootstrap complete.
I20260812 02:35:54.084089 1034 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1806e8fbbfe747018262028788ff8fcf [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1806e8fbbfe747018262028788ff8fcf" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 36667 } }
I20260812 02:35:54.084511 1034 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1806e8fbbfe747018262028788ff8fcf [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1806e8fbbfe747018262028788ff8fcf, State: Initialized, Role: FOLLOWER
I20260812 02:35:54.085072 1034 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1806e8fbbfe747018262028788ff8fcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1806e8fbbfe747018262028788ff8fcf" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 36667 } }
I20260812 02:35:54.085197 1034 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1806e8fbbfe747018262028788ff8fcf [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 02:35:54.085286 1034 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1806e8fbbfe747018262028788ff8fcf [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 02:35:54.085413 1034 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1806e8fbbfe747018262028788ff8fcf [term 2 FOLLOWER]: Advancing to term 3
I20260812 02:35:54.087589 1034 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1806e8fbbfe747018262028788ff8fcf [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1806e8fbbfe747018262028788ff8fcf" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 36667 } }
I20260812 02:35:54.087954 1034 leader_election.cc:302] T 00000000000000000000000000000000 P 1806e8fbbfe747018262028788ff8fcf [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1806e8fbbfe747018262028788ff8fcf; no voters:
I20260812 02:35:54.088337 1034 leader_election.cc:288] T 00000000000000000000000000000000 P 1806e8fbbfe747018262028788ff8fcf [CANDIDATE]: Term 3 election: Requested vote from peers
I20260812 02:35:54.088434 1039 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1806e8fbbfe747018262028788ff8fcf [term 3 FOLLOWER]: Leader election won for term 3
I20260812 02:35:54.088619 1039 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1806e8fbbfe747018262028788ff8fcf [term 3 LEADER]: Becoming Leader. State: Replica: 1806e8fbbfe747018262028788ff8fcf, State: Running, Role: LEADER
I20260812 02:35:54.089004 1039 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1806e8fbbfe747018262028788ff8fcf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1806e8fbbfe747018262028788ff8fcf" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 36667 } }
I20260812 02:35:54.089653 1034 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1806e8fbbfe747018262028788ff8fcf [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:35:54.090488 1040 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1806e8fbbfe747018262028788ff8fcf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "1806e8fbbfe747018262028788ff8fcf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1806e8fbbfe747018262028788ff8fcf" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 36667 } } }
I20260812 02:35:54.090623 1040 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1806e8fbbfe747018262028788ff8fcf [sys.catalog]: This master's current role is: LEADER
I20260812 02:35:54.091092 1041 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1806e8fbbfe747018262028788ff8fcf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1806e8fbbfe747018262028788ff8fcf. Latest consensus state: current_term: 3 leader_uuid: "1806e8fbbfe747018262028788ff8fcf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1806e8fbbfe747018262028788ff8fcf" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 36667 } } }
I20260812 02:35:54.091245 1041 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1806e8fbbfe747018262028788ff8fcf [sys.catalog]: This master's current role is: LEADER
I20260812 02:35:54.091758 1051 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 02:35:54.097743 1051 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 02:35:54.098523 1051 catalog_manager.cc:1259] Loaded cluster ID: 4bc06eec6ed3497e94bf64b1bc9c3590
I20260812 02:35:54.098595 1051 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 02:35:54.101889 1051 catalog_manager.cc:1504] Loading token signing keys...
I20260812 02:35:54.102835 1051 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 1806e8fbbfe747018262028788ff8fcf: Loaded TSK: 0
I20260812 02:35:54.102943 1051 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 1806e8fbbfe747018262028788ff8fcf: Loaded TSK: 1
F20260812 02:35:54.103771 1051 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f977c6529ad google::LogMessage::Fail() at ??:0
@ 0x7f977c65588a google::LogMessage::SendToLog() at ??:0
@ 0x7f977c6524de google::LogMessage::Flush() at ??:0
@ 0x7f977c653a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f977f31618d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f977f30ef0d kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f9786fb77bb kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f9786f9199c kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f9786f92178 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7f9786fd3df8 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f9786ff282b std::function<>::operator()() at ??:0
@ 0x7f9786f91b4f _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f9786f92f3e kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f9786f8f579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f9786fd3640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f977e8a43d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f977e8a4c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f977e8a63f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f977e895512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f977fed96db start_thread at ??:0
@ 0x7f977b15061f clone at ??:0
W20260812 02:35:54.212731 1037 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:35:54.213104 1037 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:35:54.213179 1037 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:35:54.217448 1037 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:35:54.217521 1037 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:35:54.217612 1037 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.103.1
I20260812 02:35:54.223307 1037 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.103.20:45573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.1.keytab
--principal=kudu/127.14.103.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.103.1:37887
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.103.1
--webserver_port=35679
--tserver_master_addrs=127.14.103.62:36667
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1037
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.103.1
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:35:54.224483 1037 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:35:54.225558 1037 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:35:54.236929 1096 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:35:54.236872 1089 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:35:54.237073 1093 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:35:54.237744 1037 server_base.cc:1034] running on GCE node
I20260812 02:35:54.238269 1037 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:35:54.239197 1037 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:35:54.240412 1037 hybrid_clock.cc:648] HybridClock initialized: now 1786502154240387 us; error 38 us; skew 500 ppm
Aug 12 02:35:54 dist-test-slave-2w3w krb5kdc[32148](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502154, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:35:54.244136 1037 init.cc:377] Logged in from keytab as kudu/127.14.103.1@KRBTEST.COM (short username kudu)
I20260812 02:35:54.245415 1037 webserver.cc:460] Webserver started at http://127.14.103.1:35679/ using document root <none> and password file <none>
I20260812 02:35:54.246238 1037 fs_manager.cc:359] Metadata directory not provided
I20260812 02:35:54.246348 1037 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:35:54.251894 1037 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260812 02:35:54.254834 1110 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:35:54.256506 1037 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 02:35:54.256659 1037 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/minicluster-data/ts-0/wal
uuid: "eb3490e69b2a4d72947384cae82a0d4e"
format_stamp: "Formatted at 2026-08-12 02:35:11 on dist-test-slave-2w3w"
server_key: "c676407f995241b3a413e8c7aabc506c"
server_key_iv: "e3fd61796ec1f02c47b288f367d1e9f9"
server_key_version: "encryptionkey@0"
I20260812 02:35:54.257473 1037 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:35:54.290666 1037 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:35:54.295008 1037 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:35:54.295291 1037 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:35:54.296504 1037 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:35:54.298048 1121 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260812 02:35:54.302524 1037 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260812 02:35:54.302625 1037 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.001s sys 0.000s
I20260812 02:35:54.302699 1037 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260812 02:35:54.305155 1037 ts_tablet_manager.cc:610] Registered 1 tablets
I20260812 02:35:54.305235 1037 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.001s sys 0.000s
I20260812 02:35:54.305601 1121 tablet_bootstrap.cc:492] T c057e89d50e8481fa551ffa7a5d6d166 P eb3490e69b2a4d72947384cae82a0d4e: Bootstrap starting.
I20260812 02:35:54.317353 1037 rpc_server.cc:307] RPC server started. Bound to: 127.14.103.1:37887
I20260812 02:35:54.317690 1245 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.103.1:37887 every 8 connection(s)
I20260812 02:35:54.318639 1037 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/minicluster-data/ts-0/data/info.pb
I20260812 02:35:54.320382 14748 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbep4M4/build/debug/bin/kudu as pid 1037
I20260812 02:35:54.320900 14748 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbep4M4/build/debug/bin/kudu
/tmp/dist-test-taskbep4M4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.103.2:45799
--local_ip_for_outbound_sockets=127.14.103.2
--tserver_master_addrs=127.14.103.62:36667
--webserver_port=44487
--webserver_interface=127.14.103.2
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.2.keytab
--principal=kudu/127.14.103.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.103.20:45573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 02:35:54.326200 1121 log.cc:824] T c057e89d50e8481fa551ffa7a5d6d166 P eb3490e69b2a4d72947384cae82a0d4e: Log is configured to *not* fsync() on all Append() calls
I20260812 02:35:54.336567 1121 tablet_bootstrap.cc:492] T c057e89d50e8481fa551ffa7a5d6d166 P eb3490e69b2a4d72947384cae82a0d4e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 02:35:54.337052 1121 tablet_bootstrap.cc:492] T c057e89d50e8481fa551ffa7a5d6d166 P eb3490e69b2a4d72947384cae82a0d4e: Bootstrap complete.
I20260812 02:35:54.338573 1121 ts_tablet_manager.cc:1397] T c057e89d50e8481fa551ffa7a5d6d166 P eb3490e69b2a4d72947384cae82a0d4e: Time spent bootstrapping tablet: real 0.033s user 0.029s sys 0.000s
I20260812 02:35:54.341830 1121 raft_consensus.cc:348] T c057e89d50e8481fa551ffa7a5d6d166 P eb3490e69b2a4d72947384cae82a0d4e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c0b137c9b9044e49094207590ce977e" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 45799 } } peers { permanent_uuid: "e8395e0332674f379f8c508b96e4e0c0" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 33023 } } peers { permanent_uuid: "eb3490e69b2a4d72947384cae82a0d4e" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 37887 } }
I20260812 02:35:54.342384 1121 raft_consensus.cc:729] T c057e89d50e8481fa551ffa7a5d6d166 P eb3490e69b2a4d72947384cae82a0d4e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb3490e69b2a4d72947384cae82a0d4e, State: Initialized, Role: FOLLOWER
I20260812 02:35:54.342907 1121 consensus_queue.cc:260] T c057e89d50e8481fa551ffa7a5d6d166 P eb3490e69b2a4d72947384cae82a0d4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c0b137c9b9044e49094207590ce977e" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 45799 } } peers { permanent_uuid: "e8395e0332674f379f8c508b96e4e0c0" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 33023 } } peers { permanent_uuid: "eb3490e69b2a4d72947384cae82a0d4e" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 37887 } }
I20260812 02:35:54.344022 1121 ts_tablet_manager.cc:1428] T c057e89d50e8481fa551ffa7a5d6d166 P eb3490e69b2a4d72947384cae82a0d4e: Time spent starting tablet: real 0.005s user 0.008s sys 0.000s
W20260812 02:35:54.363842 1248 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:35:54.321019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:36667 (local address 127.14.103.1:33381)
0812 02:35:54.322659 (+ 1640us) negotiation.cc:107] Waiting for socket to connect
0812 02:35:54.322733 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0812 02:35:54.323907 (+ 1174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:35:54.362785 (+ 38878us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.62:36667: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":1336,"thread_start_us":182,"threads_started":1}
W20260812 02:35:54.364485 1246 heartbeater.cc:644] Failed to heartbeat to 127.14.103.62:36667 (0 consecutive failures): Network error: Failed to ping master at 127.14.103.62:36667: Client connection negotiation failed: client connection to 127.14.103.62:36667: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260812 02:35:54.365309 1248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:54.365009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:36667 (local address 127.14.103.1:55051)
0812 02:35:54.365164 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0812 02:35:54.365241 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.62:36667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260812 02:35:54.366037 1248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:54.365778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:36667 (local address 127.14.103.1:39381)
0812 02:35:54.365886 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0812 02:35:54.365936 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.62:36667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260812 02:35:54.366232 1246 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260812 02:35:54.448181 1249 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:35:54.448493 1249 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:35:54.448542 1249 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:35:54.452812 1249 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:35:54.452885 1249 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:35:54.452946 1249 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.103.2
I20260812 02:35:54.458478 1249 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.103.20:45573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.2.keytab
--principal=kudu/127.14.103.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.103.2:45799
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.103.2
--webserver_port=44487
--tserver_master_addrs=127.14.103.62:36667
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1249
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.103.2
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:35:54.459641 1249 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:35:54.460692 1249 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:35:54.470974 1256 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:35:54.471076 1257 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:35:54.471060 1249 server_base.cc:1034] running on GCE node
W20260812 02:35:54.470974 1259 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:35:54.471693 1249 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:35:54.472368 1249 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:35:54.473543 1249 hybrid_clock.cc:648] HybridClock initialized: now 1786502154473527 us; error 29 us; skew 500 ppm
Aug 12 02:35:54 dist-test-slave-2w3w krb5kdc[32148](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502154, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:35:54.476598 1249 init.cc:377] Logged in from keytab as kudu/127.14.103.2@KRBTEST.COM (short username kudu)
I20260812 02:35:54.477521 1249 webserver.cc:460] Webserver started at http://127.14.103.2:44487/ using document root <none> and password file <none>
I20260812 02:35:54.478132 1249 fs_manager.cc:359] Metadata directory not provided
I20260812 02:35:54.478202 1249 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:35:54.482503 1249 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260812 02:35:54.484783 1268 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:35:54.486169 1249 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260812 02:35:54.486330 1249 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/minicluster-data/ts-1/wal
uuid: "4c0b137c9b9044e49094207590ce977e"
format_stamp: "Formatted at 2026-08-12 02:35:11 on dist-test-slave-2w3w"
server_key: "f4f69eb05b1f444046bfdcdd6e2e1ce8"
server_key_iv: "793d2d7b936db6cd2822f67e83dc527e"
server_key_version: "encryptionkey@0"
I20260812 02:35:54.487139 1249 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:35:54.516773 1249 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:35:54.520179 1249 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:35:54.520442 1249 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:35:54.521087 1249 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:35:54.521958 1249 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:35:54.522032 1249 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:35:54.522104 1249 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:35:54.522149 1249 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:35:54.531967 1249 rpc_server.cc:307] RPC server started. Bound to: 127.14.103.2:45799
I20260812 02:35:54.532070 1400 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.103.2:45799 every 8 connection(s)
I20260812 02:35:54.533057 1249 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/minicluster-data/ts-1/data/info.pb
I20260812 02:35:54.537437 1403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:54.535447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:36667 (local address 127.14.103.2:46987)
0812 02:35:54.535970 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0812 02:35:54.536086 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.62:36667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":125,"threads_started":1}
W20260812 02:35:54.538034 1401 heartbeater.cc:644] Failed to heartbeat to 127.14.103.62:36667 (0 consecutive failures): Network error: Failed to ping master at 127.14.103.62:36667: Client connection negotiation failed: client connection to 127.14.103.62:36667: connect: Connection refused (error 111)
I20260812 02:35:54.538760 1403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:54.538525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:36667 (local address 127.14.103.2:52405)
0812 02:35:54.538646 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0812 02:35:54.538699 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.62:36667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260812 02:35:54.539532 1403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:54.539325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:36667 (local address 127.14.103.2:45429)
0812 02:35:54.539438 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0812 02:35:54.539483 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.62:36667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260812 02:35:54.539778 1401 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260812 02:35:54.542539 14748 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbep4M4/build/debug/bin/kudu as pid 1249
I20260812 02:35:54.543107 14748 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbep4M4/build/debug/bin/kudu
/tmp/dist-test-taskbep4M4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.103.3:33023
--local_ip_for_outbound_sockets=127.14.103.3
--tserver_master_addrs=127.14.103.62:36667
--webserver_port=44141
--webserver_interface=127.14.103.3
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.3.keytab
--principal=kudu/127.14.103.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.103.20:45573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:35:54.662894 1404 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:35:54.663252 1404 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:35:54.663353 1404 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:35:54.667666 1404 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:35:54.667765 1404 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:35:54.667866 1404 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.103.3
I20260812 02:35:54.673455 1404 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.103.20:45573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.3.keytab
--principal=kudu/127.14.103.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.103.3:33023
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.103.3
--webserver_port=44141
--tserver_master_addrs=127.14.103.62:36667
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1404
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.103.3
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:35:54.674579 1404 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:35:54.675649 1404 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:35:54.686334 1413 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:35:54.686313 1410 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:35:54.686520 1409 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:35:54.686571 1404 server_base.cc:1034] running on GCE node
I20260812 02:35:54.687285 1404 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:35:54.688025 1404 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:35:54.689239 1404 hybrid_clock.cc:648] HybridClock initialized: now 1786502154689218 us; error 32 us; skew 500 ppm
Aug 12 02:35:54 dist-test-slave-2w3w krb5kdc[32148](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502154, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:35:54.692802 1404 init.cc:377] Logged in from keytab as kudu/127.14.103.3@KRBTEST.COM (short username kudu)
I20260812 02:35:54.693784 1404 webserver.cc:460] Webserver started at http://127.14.103.3:44141/ using document root <none> and password file <none>
I20260812 02:35:54.694352 1404 fs_manager.cc:359] Metadata directory not provided
I20260812 02:35:54.694415 1404 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:35:54.698601 1404 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260812 02:35:54.701014 1426 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:35:54.702383 1404 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260812 02:35:54.702544 1404 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/minicluster-data/ts-2/wal
uuid: "e8395e0332674f379f8c508b96e4e0c0"
format_stamp: "Formatted at 2026-08-12 02:35:11 on dist-test-slave-2w3w"
server_key: "985211a092ad46521774423ece2bee0a"
server_key_iv: "a128ee492a442ed5171334ed7cbc8269"
server_key_version: "encryptionkey@0"
I20260812 02:35:54.703199 1404 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:35:54.714033 1404 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:35:54.717605 1404 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:35:54.717898 1404 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:35:54.718772 1404 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:35:54.719940 1404 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:35:54.719997 1404 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:35:54.720043 1404 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:35:54.720103 1404 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:35:54.730036 1404 rpc_server.cc:307] RPC server started. Bound to: 127.14.103.3:33023
I20260812 02:35:54.730125 1559 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.103.3:33023 every 8 connection(s)
I20260812 02:35:54.731185 1404 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786501898713699-14748-0/minicluster-data/ts-2/data/info.pb
I20260812 02:35:54.731724 14748 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbep4M4/build/debug/bin/kudu as pid 1404
I20260812 02:35:54.735492 1563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:54.733573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:36667 (local address 127.14.103.3:55945)
0812 02:35:54.734059 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0812 02:35:54.734164 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.62:36667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":117,"threads_started":1}
W20260812 02:35:54.736030 1560 heartbeater.cc:644] Failed to heartbeat to 127.14.103.62:36667 (0 consecutive failures): Network error: Failed to ping master at 127.14.103.62:36667: Client connection negotiation failed: client connection to 127.14.103.62:36667: connect: Connection refused (error 111)
I20260812 02:35:54.736717 1563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:54.736470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:36667 (local address 127.14.103.3:59639)
0812 02:35:54.736599 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0812 02:35:54.736653 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.62:36667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260812 02:35:54.737491 1563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:54.737280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:36667 (local address 127.14.103.3:43269)
0812 02:35:54.737392 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0812 02:35:54.737437 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.62:36667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260812 02:35:54.737722 1560 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260812 02:35:55.367559 1564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:55.366954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:36667 (local address 127.14.103.1:49035)
0812 02:35:55.367340 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0812 02:35:55.367406 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.62:36667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":97,"threads_started":1}
I20260812 02:35:55.530288 1565 raft_consensus.cc:482] T c057e89d50e8481fa551ffa7a5d6d166 P eb3490e69b2a4d72947384cae82a0d4e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 02:35:55.530537 1565 raft_consensus.cc:504] T c057e89d50e8481fa551ffa7a5d6d166 P eb3490e69b2a4d72947384cae82a0d4e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c0b137c9b9044e49094207590ce977e" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 45799 } } peers { permanent_uuid: "e8395e0332674f379f8c508b96e4e0c0" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 33023 } } peers { permanent_uuid: "eb3490e69b2a4d72947384cae82a0d4e" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 37887 } }
I20260812 02:35:55.531816 1565 leader_election.cc:288] T c057e89d50e8481fa551ffa7a5d6d166 P eb3490e69b2a4d72947384cae82a0d4e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c0b137c9b9044e49094207590ce977e (127.14.103.2:45799), e8395e0332674f379f8c508b96e4e0c0 (127.14.103.3:33023)
I20260812 02:35:55.540799 1569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:55.540328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:36667 (local address 127.14.103.2:43847)
0812 02:35:55.540627 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0812 02:35:55.540682 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.62:36667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":108,"threads_started":1}
Aug 12 02:35:55 dist-test-slave-2w3w krb5kdc[32148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502154, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.1@KRBTEST.COM for kudu/127.14.103.3@KRBTEST.COM
Aug 12 02:35:55 dist-test-slave-2w3w krb5kdc[32148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502154, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.1@KRBTEST.COM for kudu/127.14.103.2@KRBTEST.COM
I20260812 02:35:55.553100 1567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:55.532386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.1:51035 (local address 127.14.103.3:33023)
0812 02:35:55.532751 (+ 365us) server_negotiation.cc:206] Beginning negotiation
0812 02:35:55.532766 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0812 02:35:55.533077 (+ 311us) server_negotiation.cc:407] Connection header received
0812 02:35:55.533960 (+ 883us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:35:55.533968 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:35:55.534360 (+ 392us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:35:55.534640 (+ 280us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:35:55.535853 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:35:55.537276 (+ 1423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:35:55.538149 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:35:55.538616 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:35:55.538787 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:35:55.548173 (+ 9386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:35:55.548180 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:35:55.548194 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:35:55.548229 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:35:55.550211 (+ 1982us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:35:55.550858 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:35:55.550871 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:35:55.550875 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:35:55.551080 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:35:55.551447 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:35:55.551452 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:35:55.551453 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:35:55.551920 (+ 467us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:35:55.552068 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0812 02:35:55.552636 (+ 568us) server_negotiation.cc:299] Negotiation successful
0812 02:35:55.552904 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256,"thread_start_us":74,"threads_started":1}
I20260812 02:35:55.553086 1568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:55.532110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.3:33023 (local address 127.14.103.1:51035)
0812 02:35:55.532950 (+ 840us) negotiation.cc:107] Waiting for socket to connect
0812 02:35:55.532962 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 02:35:55.533086 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:35:55.535020 (+ 1934us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:35:55.535023 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:35:55.535222 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:35:55.535656 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:35:55.535674 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:35:55.537407 (+ 1733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:35:55.537414 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:35:55.538017 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:35:55.538026 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:35:55.538756 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:35:55.538759 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:35:55.539025 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:35:55.545937 (+ 6912us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:35:55.545959 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:35:55.547830 (+ 1871us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:35:55.550364 (+ 2534us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:35:55.550370 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:35:55.550385 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:35:55.550726 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:35:55.551183 (+ 457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:35:55.551186 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:35:55.551188 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:35:55.551342 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:35:55.552043 (+ 701us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:35:55.552050 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:35:55.552320 (+ 270us) client_negotiation.cc:769] Sending connection context
0812 02:35:55.552520 (+ 200us) client_negotiation.cc:240] Negotiation successful
0812 02:35:55.552888 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":777,"mutex_wait_us":613,"spinlock_wait_cycles":10624,"thread_start_us":105,"threads_started":1}
I20260812 02:35:55.553133 1564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:55.532102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.2:45799 (local address 127.14.103.1:45601)
0812 02:35:55.532372 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0812 02:35:55.532389 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 02:35:55.532508 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:35:55.534478 (+ 1970us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:35:55.534484 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:35:55.535044 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:35:55.535654 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:35:55.535670 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:35:55.537094 (+ 1424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:35:55.537098 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:35:55.538022 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:35:55.538032 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:35:55.538757 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:35:55.538760 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:35:55.539036 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:35:55.545936 (+ 6900us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:35:55.545964 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:35:55.548124 (+ 2160us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:35:55.550452 (+ 2328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:35:55.550458 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:35:55.550460 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:35:55.550810 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:35:55.551279 (+ 469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:35:55.551283 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:35:55.551285 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:35:55.551408 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:35:55.552066 (+ 658us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:35:55.552069 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:35:55.552319 (+ 250us) client_negotiation.cc:769] Sending connection context
0812 02:35:55.552520 (+ 201us) client_negotiation.cc:240] Negotiation successful
0812 02:35:55.552974 (+ 454us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"mutex_wait_us":226,"spinlock_wait_cycles":36480}
I20260812 02:35:55.553500 1566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:55.532267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.1:45601 (local address 127.14.103.2:45799)
0812 02:35:55.532651 (+ 384us) server_negotiation.cc:206] Beginning negotiation
0812 02:35:55.532665 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0812 02:35:55.532709 (+ 44us) server_negotiation.cc:407] Connection header received
0812 02:35:55.533612 (+ 903us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:35:55.533621 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:35:55.534073 (+ 452us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:35:55.534403 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:35:55.535864 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:35:55.536933 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:35:55.538159 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:35:55.538617 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:35:55.538796 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:35:55.548371 (+ 9575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:35:55.548376 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:35:55.548389 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:35:55.548423 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:35:55.550305 (+ 1882us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:35:55.551010 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:35:55.551023 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:35:55.551028 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:35:55.551167 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:35:55.551517 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:35:55.551521 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:35:55.551525 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:35:55.551948 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:35:55.552101 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0812 02:35:55.552642 (+ 541us) server_negotiation.cc:299] Negotiation successful
0812 02:35:55.552954 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":75,"threads_started":1}
W20260812 02:35:55.556434 1114 leader_election.cc:341] T c057e89d50e8481fa551ffa7a5d6d166 P eb3490e69b2a4d72947384cae82a0d4e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4c0b137c9b9044e49094207590ce977e (127.14.103.2:45799): Not found: Tablet not found: c057e89d50e8481fa551ffa7a5d6d166
W20260812 02:35:55.556596 1112 leader_election.cc:341] T c057e89d50e8481fa551ffa7a5d6d166 P eb3490e69b2a4d72947384cae82a0d4e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e8395e0332674f379f8c508b96e4e0c0 (127.14.103.3:33023): Not found: Tablet not found: c057e89d50e8481fa551ffa7a5d6d166
I20260812 02:35:55.556720 1112 leader_election.cc:302] T c057e89d50e8481fa551ffa7a5d6d166 P eb3490e69b2a4d72947384cae82a0d4e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb3490e69b2a4d72947384cae82a0d4e; no voters: 4c0b137c9b9044e49094207590ce977e, e8395e0332674f379f8c508b96e4e0c0
I20260812 02:35:55.557040 1565 raft_consensus.cc:2726] T c057e89d50e8481fa551ffa7a5d6d166 P eb3490e69b2a4d72947384cae82a0d4e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 02:35:55.738848 1570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:55.738366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:36667 (local address 127.14.103.3:57795)
0812 02:35:55.738685 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0812 02:35:55.738746 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.62:36667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":88,"threads_started":1}
I20260812 02:35:56.368975 1573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:56.368514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:36667 (local address 127.14.103.1:51215)
0812 02:35:56.368806 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0812 02:35:56.368876 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.62:36667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":82,"threads_started":1}
I20260812 02:35:56.542299 1574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:56.541822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:36667 (local address 127.14.103.2:42681)
0812 02:35:56.542135 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0812 02:35:56.542200 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.62:36667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":90,"threads_started":1}
I20260812 02:35:56.740257 1575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:56.739788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:36667 (local address 127.14.103.3:56061)
0812 02:35:56.740081 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0812 02:35:56.740145 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.62:36667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":81,"threads_started":1}
I20260812 02:35:57.370362 1576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:57.369895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:36667 (local address 127.14.103.1:51991)
0812 02:35:57.370198 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0812 02:35:57.370262 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.62:36667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":105,"threads_started":1}
I20260812 02:35:57.543747 1579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:57.543271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:36667 (local address 127.14.103.2:58059)
0812 02:35:57.543579 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0812 02:35:57.543644 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.62:36667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":85,"threads_started":1}
I20260812 02:35:57.725571 1580 raft_consensus.cc:482] T c057e89d50e8481fa551ffa7a5d6d166 P eb3490e69b2a4d72947384cae82a0d4e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 02:35:57.725721 1580 raft_consensus.cc:504] T c057e89d50e8481fa551ffa7a5d6d166 P eb3490e69b2a4d72947384cae82a0d4e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c0b137c9b9044e49094207590ce977e" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 45799 } } peers { permanent_uuid: "e8395e0332674f379f8c508b96e4e0c0" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 33023 } } peers { permanent_uuid: "eb3490e69b2a4d72947384cae82a0d4e" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 37887 } }
I20260812 02:35:57.726172 1580 leader_election.cc:288] T c057e89d50e8481fa551ffa7a5d6d166 P eb3490e69b2a4d72947384cae82a0d4e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c0b137c9b9044e49094207590ce977e (127.14.103.2:45799), e8395e0332674f379f8c508b96e4e0c0 (127.14.103.3:33023)
W20260812 02:35:57.727133 1114 leader_election.cc:341] T c057e89d50e8481fa551ffa7a5d6d166 P eb3490e69b2a4d72947384cae82a0d4e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4c0b137c9b9044e49094207590ce977e (127.14.103.2:45799): Not found: Tablet not found: c057e89d50e8481fa551ffa7a5d6d166
W20260812 02:35:57.727237 1112 leader_election.cc:341] T c057e89d50e8481fa551ffa7a5d6d166 P eb3490e69b2a4d72947384cae82a0d4e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e8395e0332674f379f8c508b96e4e0c0 (127.14.103.3:33023): Not found: Tablet not found: c057e89d50e8481fa551ffa7a5d6d166
I20260812 02:35:57.727349 1112 leader_election.cc:302] T c057e89d50e8481fa551ffa7a5d6d166 P eb3490e69b2a4d72947384cae82a0d4e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb3490e69b2a4d72947384cae82a0d4e; no voters: 4c0b137c9b9044e49094207590ce977e, e8395e0332674f379f8c508b96e4e0c0
I20260812 02:35:57.727537 1580 raft_consensus.cc:2726] T c057e89d50e8481fa551ffa7a5d6d166 P eb3490e69b2a4d72947384cae82a0d4e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 02:35:57.741622 1582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:57.741156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:36667 (local address 127.14.103.3:54295)
0812 02:35:57.741461 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0812 02:35:57.741525 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.62:36667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":84,"threads_started":1}
I20260812 02:35:58.371904 1583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:58.371367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:36667 (local address 127.14.103.1:50589)
0812 02:35:58.371696 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0812 02:35:58.371760 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.62:36667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":94,"threads_started":1}
I20260812 02:35:58.545153 1585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:58.544632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:36667 (local address 127.14.103.2:40235)
0812 02:35:58.544952 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0812 02:35:58.545031 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.62:36667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":103,"threads_started":1}
I20260812 02:35:58.743129 1586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:58.742577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:36667 (local address 127.14.103.3:54241)
0812 02:35:58.742865 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0812 02:35:58.742942 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.62:36667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":80,"threads_started":1}
I20260812 02:35:59.373577 1587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:59.372972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:36667 (local address 127.14.103.1:37825)
0812 02:35:59.373319 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0812 02:35:59.373447 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.62:36667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":99,"threads_started":1}
I20260812 02:35:59.546540 1589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:59.546072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:36667 (local address 127.14.103.2:50423)
0812 02:35:59.546362 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0812 02:35:59.546440 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.62:36667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":81,"threads_started":1}
I20260812 02:35:59.744532 1590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:35:59.744082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:36667 (local address 127.14.103.3:39773)
0812 02:35:59.744357 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0812 02:35:59.744419 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.62:36667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":80,"threads_started":1}
I20260812 02:36:00.126482 1591 raft_consensus.cc:482] T c057e89d50e8481fa551ffa7a5d6d166 P eb3490e69b2a4d72947384cae82a0d4e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 02:36:00.126663 1591 raft_consensus.cc:504] T c057e89d50e8481fa551ffa7a5d6d166 P eb3490e69b2a4d72947384cae82a0d4e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c0b137c9b9044e49094207590ce977e" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 45799 } } peers { permanent_uuid: "e8395e0332674f379f8c508b96e4e0c0" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 33023 } } peers { permanent_uuid: "eb3490e69b2a4d72947384cae82a0d4e" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 37887 } }
I20260812 02:36:00.127279 1591 leader_election.cc:288] T c057e89d50e8481fa551ffa7a5d6d166 P eb3490e69b2a4d72947384cae82a0d4e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c0b137c9b9044e49094207590ce977e (127.14.103.2:45799), e8395e0332674f379f8c508b96e4e0c0 (127.14.103.3:33023)
W20260812 02:36:00.128039 1114 leader_election.cc:341] T c057e89d50e8481fa551ffa7a5d6d166 P eb3490e69b2a4d72947384cae82a0d4e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4c0b137c9b9044e49094207590ce977e (127.14.103.2:45799): Not found: Tablet not found: c057e89d50e8481fa551ffa7a5d6d166
W20260812 02:36:00.128165 1112 leader_election.cc:341] T c057e89d50e8481fa551ffa7a5d6d166 P eb3490e69b2a4d72947384cae82a0d4e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e8395e0332674f379f8c508b96e4e0c0 (127.14.103.3:33023): Not found: Tablet not found: c057e89d50e8481fa551ffa7a5d6d166
I20260812 02:36:00.128255 1112 leader_election.cc:302] T c057e89d50e8481fa551ffa7a5d6d166 P eb3490e69b2a4d72947384cae82a0d4e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb3490e69b2a4d72947384cae82a0d4e; no voters: 4c0b137c9b9044e49094207590ce977e, e8395e0332674f379f8c508b96e4e0c0
I20260812 02:36:00.128412 1591 raft_consensus.cc:2726] T c057e89d50e8481fa551ffa7a5d6d166 P eb3490e69b2a4d72947384cae82a0d4e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 02:36:00.375170 1592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:00.374567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:36667 (local address 127.14.103.1:41903)
0812 02:36:00.374922 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0812 02:36:00.375064 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.62:36667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":83,"threads_started":1}
I20260812 02:36:00.547978 1593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:00.547470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:36667 (local address 127.14.103.2:40461)
0812 02:36:00.547767 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0812 02:36:00.547836 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.62:36667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":77,"threads_started":1}
I20260812 02:36:00.745980 1594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:00.745482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:36667 (local address 127.14.103.3:59995)
0812 02:36:00.745804 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0812 02:36:00.745871 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.62:36667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":99,"threads_started":1}
I20260812 02:36:01.376803 1595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:01.376304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:36667 (local address 127.14.103.1:33649)
0812 02:36:01.376633 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0812 02:36:01.376703 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.62:36667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":86,"threads_started":1}
I20260812 02:36:01.549335 1596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:01.548845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:36667 (local address 127.14.103.2:53855)
0812 02:36:01.549180 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0812 02:36:01.549234 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.62:36667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":100,"threads_started":1}
I20260812 02:36:01.747691 1598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:01.746954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:36667 (local address 127.14.103.3:54039)
0812 02:36:01.747438 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0812 02:36:01.747586 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.62:36667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":184,"threads_started":1}
I20260812 02:36:02.378180 1599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:02.377688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:36667 (local address 127.14.103.1:33391)
0812 02:36:02.378020 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0812 02:36:02.378083 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.62:36667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":106,"threads_started":1}
I20260812 02:36:02.550820 1600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:02.550313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:36667 (local address 127.14.103.2:39739)
0812 02:36:02.550629 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0812 02:36:02.550697 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.62:36667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":88,"threads_started":1}
I20260812 02:36:02.749351 1601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:02.748746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:36667 (local address 127.14.103.3:55915)
0812 02:36:02.749138 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0812 02:36:02.749224 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.62:36667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":142,"threads_started":1}
I20260812 02:36:03.379765 1603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:03.379277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:36667 (local address 127.14.103.1:53373)
0812 02:36:03.379601 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0812 02:36:03.379667 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.62:36667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":139,"threads_started":1}
I20260812 02:36:03.552239 1604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:03.551741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:36667 (local address 127.14.103.2:49193)
0812 02:36:03.552060 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0812 02:36:03.552140 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.62:36667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":96,"threads_started":1}
I20260812 02:36:03.750793 1606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:03.750323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:36667 (local address 127.14.103.3:58539)
0812 02:36:03.750627 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0812 02:36:03.750691 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.62:36667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":89,"threads_started":1}
I20260812 02:36:04.381531 1607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:04.380997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:36667 (local address 127.14.103.1:52317)
0812 02:36:04.381316 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0812 02:36:04.381413 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.62:36667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":90,"threads_started":1}
I20260812 02:36:04.553653 1609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:04.553147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:36667 (local address 127.14.103.2:41467)
0812 02:36:04.553464 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0812 02:36:04.553530 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.62:36667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":82,"threads_started":1}
I20260812 02:36:04.584894 1610 raft_consensus.cc:482] T c057e89d50e8481fa551ffa7a5d6d166 P eb3490e69b2a4d72947384cae82a0d4e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 02:36:04.585124 1610 raft_consensus.cc:504] T c057e89d50e8481fa551ffa7a5d6d166 P eb3490e69b2a4d72947384cae82a0d4e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c0b137c9b9044e49094207590ce977e" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 45799 } } peers { permanent_uuid: "e8395e0332674f379f8c508b96e4e0c0" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 33023 } } peers { permanent_uuid: "eb3490e69b2a4d72947384cae82a0d4e" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 37887 } }
I20260812 02:36:04.585624 1610 leader_election.cc:288] T c057e89d50e8481fa551ffa7a5d6d166 P eb3490e69b2a4d72947384cae82a0d4e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c0b137c9b9044e49094207590ce977e (127.14.103.2:45799), e8395e0332674f379f8c508b96e4e0c0 (127.14.103.3:33023)
W20260812 02:36:04.586542 1112 leader_election.cc:341] T c057e89d50e8481fa551ffa7a5d6d166 P eb3490e69b2a4d72947384cae82a0d4e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e8395e0332674f379f8c508b96e4e0c0 (127.14.103.3:33023): Not found: Tablet not found: c057e89d50e8481fa551ffa7a5d6d166
W20260812 02:36:04.586653 1114 leader_election.cc:341] T c057e89d50e8481fa551ffa7a5d6d166 P eb3490e69b2a4d72947384cae82a0d4e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4c0b137c9b9044e49094207590ce977e (127.14.103.2:45799): Not found: Tablet not found: c057e89d50e8481fa551ffa7a5d6d166
I20260812 02:36:04.586716 1114 leader_election.cc:302] T c057e89d50e8481fa551ffa7a5d6d166 P eb3490e69b2a4d72947384cae82a0d4e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb3490e69b2a4d72947384cae82a0d4e; no voters: 4c0b137c9b9044e49094207590ce977e, e8395e0332674f379f8c508b96e4e0c0
I20260812 02:36:04.586872 1610 raft_consensus.cc:2726] T c057e89d50e8481fa551ffa7a5d6d166 P eb3490e69b2a4d72947384cae82a0d4e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 02:36:04.752251 1611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:04.751721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:36667 (local address 127.14.103.3:54769)
0812 02:36:04.752075 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0812 02:36:04.752150 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.62:36667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":117,"threads_started":1}
I20260812 02:36:05.383172 1612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:05.382577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:36667 (local address 127.14.103.1:55869)
0812 02:36:05.382901 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0812 02:36:05.383044 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.62:36667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":85,"threads_started":1}
I20260812 02:36:05.555109 1613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:05.554521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:36667 (local address 127.14.103.2:45643)
0812 02:36:05.554882 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0812 02:36:05.554945 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.62:36667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":93,"threads_started":1}
I20260812 02:36:05.753878 1614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:05.753238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:36667 (local address 127.14.103.3:58011)
0812 02:36:05.753672 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0812 02:36:05.753752 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.62:36667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":118,"threads_started":1}
I20260812 02:36:06.384580 1615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:06.384124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:36667 (local address 127.14.103.1:45617)
0812 02:36:06.384405 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0812 02:36:06.384478 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.62:36667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":85,"threads_started":1}
I20260812 02:36:06.556417 1616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:06.555959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:36667 (local address 127.14.103.2:47531)
0812 02:36:06.556240 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0812 02:36:06.556311 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.62:36667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":83,"threads_started":1}
I20260812 02:36:06.755306 1617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:06.754657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:36667 (local address 127.14.103.3:57033)
0812 02:36:06.754947 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0812 02:36:06.755147 (+ 200us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.62:36667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":89,"threads_started":1}
I20260812 02:36:07.386065 1618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:07.385511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:36667 (local address 127.14.103.1:34801)
0812 02:36:07.385880 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0812 02:36:07.385957 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.62:36667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":88,"threads_started":1}
I20260812 02:36:07.557677 1619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:07.557257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:36667 (local address 127.14.103.2:47403)
0812 02:36:07.557519 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0812 02:36:07.557577 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.62:36667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":85,"threads_started":1}
I20260812 02:36:07.756552 1621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:07.756091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:36667 (local address 127.14.103.3:54615)
0812 02:36:07.756393 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0812 02:36:07.756453 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.62:36667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":82,"threads_started":1}
I20260812 02:36:08.387673 1624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:08.387197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:36667 (local address 127.14.103.1:50899)
0812 02:36:08.387509 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0812 02:36:08.387576 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.62:36667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":87,"threads_started":1}
I20260812 02:36:08.559144 1625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:08.558636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:36667 (local address 127.14.103.2:54685)
0812 02:36:08.558909 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0812 02:36:08.559043 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.62:36667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":80,"threads_started":1}
I20260812 02:36:08.667263 1626 raft_consensus.cc:482] T c057e89d50e8481fa551ffa7a5d6d166 P eb3490e69b2a4d72947384cae82a0d4e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 02:36:08.667457 1626 raft_consensus.cc:504] T c057e89d50e8481fa551ffa7a5d6d166 P eb3490e69b2a4d72947384cae82a0d4e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c0b137c9b9044e49094207590ce977e" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 45799 } } peers { permanent_uuid: "e8395e0332674f379f8c508b96e4e0c0" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 33023 } } peers { permanent_uuid: "eb3490e69b2a4d72947384cae82a0d4e" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 37887 } }
I20260812 02:36:08.667968 1626 leader_election.cc:288] T c057e89d50e8481fa551ffa7a5d6d166 P eb3490e69b2a4d72947384cae82a0d4e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c0b137c9b9044e49094207590ce977e (127.14.103.2:45799), e8395e0332674f379f8c508b96e4e0c0 (127.14.103.3:33023)
W20260812 02:36:08.668823 1112 leader_election.cc:341] T c057e89d50e8481fa551ffa7a5d6d166 P eb3490e69b2a4d72947384cae82a0d4e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e8395e0332674f379f8c508b96e4e0c0 (127.14.103.3:33023): Not found: Tablet not found: c057e89d50e8481fa551ffa7a5d6d166
W20260812 02:36:08.668962 1114 leader_election.cc:341] T c057e89d50e8481fa551ffa7a5d6d166 P eb3490e69b2a4d72947384cae82a0d4e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4c0b137c9b9044e49094207590ce977e (127.14.103.2:45799): Not found: Tablet not found: c057e89d50e8481fa551ffa7a5d6d166
I20260812 02:36:08.669075 1114 leader_election.cc:302] T c057e89d50e8481fa551ffa7a5d6d166 P eb3490e69b2a4d72947384cae82a0d4e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb3490e69b2a4d72947384cae82a0d4e; no voters: 4c0b137c9b9044e49094207590ce977e, e8395e0332674f379f8c508b96e4e0c0
I20260812 02:36:08.669239 1626 raft_consensus.cc:2726] T c057e89d50e8481fa551ffa7a5d6d166 P eb3490e69b2a4d72947384cae82a0d4e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 02:36:08.757893 1628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:08.757457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:36667 (local address 127.14.103.3:51299)
0812 02:36:08.757734 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0812 02:36:08.757794 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.62:36667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":87,"threads_started":1}
I20260812 02:36:09.389166 1629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:09.388725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:36667 (local address 127.14.103.1:34149)
0812 02:36:09.389017 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0812 02:36:09.389072 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.62:36667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":111,"threads_started":1}
I20260812 02:36:09.560665 1630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:09.560221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:36667 (local address 127.14.103.2:44175)
0812 02:36:09.560503 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0812 02:36:09.560566 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.62:36667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":78,"threads_started":1}
I20260812 02:36:09.742192 14748 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbep4M4/build/debug/bin/kudu with pid 1037
I20260812 02:36:09.750856 14748 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbep4M4/build/debug/bin/kudu with pid 1249
I20260812 02:36:09.759367 1631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:09.758853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:36667 (local address 127.14.103.3:48983)
0812 02:36:09.759200 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0812 02:36:09.759268 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.62:36667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":170,"threads_started":1}
I20260812 02:36:09.760182 14748 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbep4M4/build/debug/bin/kudu with pid 1404
2026-08-12T02:36:09Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (60109 ms)
[----------] 30 tests from SecurityITest (271003 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501898713699-14748-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 02:36:09 dist-test-slave-2w3w krb5kdc[1634](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 02:36:09 dist-test-slave-2w3w krb5kdc[1634](info): set up 2 sockets
Aug 12 02:36:09 dist-test-slave-2w3w krb5kdc[1634](info): commencing operation
krb5kdc: starting...
W20260812 02:36:10.910459 14748 mini_kdc.cc:121] Time spent starting KDC: real 1.124s user 0.002s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501898713699-14748-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501898713699-14748-0/krb5kdc/test-user.keytab.
Aug 12 02:36:11 dist-test-slave-2w3w krb5kdc[1634](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502171, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T02:36:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T02:36:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.103.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.103.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.103.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.62.keytab.
Entry for principal kudu/127.14.103.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.62.keytab.
I20260812 02:36:11.055212 14748 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbep4M4/build/debug/bin/kudu
/tmp/dist-test-taskbep4M4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501898713699-14748-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501898713699-14748-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501898713699-14748-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501898713699-14748-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.103.62:44451
--webserver_interface=127.14.103.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.62.keytab
--principal=kudu/127.14.103.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.103.20:43843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.103.62:44451
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501898713699-14748-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501898713699-14748-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501898713699-14748-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:36:11.171129 1650 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:36:11.171492 1650 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:36:11.171588 1650 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:36:11.176054 1650 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:36:11.176172 1650 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:36:11.176213 1650 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:36:11.176282 1650 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:36:11.176337 1650 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 02:36:11.182114 1650 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.103.20:43843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501898713699-14748-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501898713699-14748-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.103.62:44451
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.62.keytab
--principal=kudu/127.14.103.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.103.62:44451
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501898713699-14748-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.103.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1650
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501898713699-14748-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:36:11.183398 1650 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:36:11.184474 1650 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:36:11.193929 1656 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:36:11.193960 1655 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:36:11.194080 1650 server_base.cc:1034] running on GCE node
W20260812 02:36:11.193923 1658 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:36:11.194907 1650 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:36:11.196171 1650 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:36:11.197371 1650 hybrid_clock.cc:648] HybridClock initialized: now 1786502171197352 us; error 33 us; skew 500 ppm
Aug 12 02:36:11 dist-test-slave-2w3w krb5kdc[1634](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502171, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:36:11.200227 1650 init.cc:377] Logged in from keytab as kudu/127.14.103.62@KRBTEST.COM (short username kudu)
I20260812 02:36:11.201207 1650 webserver.cc:460] Webserver started at http://127.14.103.62:38187/ using document root <none> and password file <none>
I20260812 02:36:11.201790 1650 fs_manager.cc:359] Metadata directory not provided
I20260812 02:36:11.201889 1650 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:36:11.202135 1650 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:36:11.204326 1650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501898713699-14748-0/minicluster-data/master-0/data/instance:
uuid: "185ba7148bd040a4b144023f244d5e1d"
format_stamp: "Formatted at 2026-08-12 02:36:11 on dist-test-slave-2w3w"
server_key: "35b269d39a440a67a896716e119a0a48"
server_key_iv: "0f0a7a4dd421b8e2f19581ab9248bc09"
server_key_version: "encryptionkey@0"
I20260812 02:36:11.204938 1650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501898713699-14748-0/minicluster-data/master-0/wal/instance:
uuid: "185ba7148bd040a4b144023f244d5e1d"
format_stamp: "Formatted at 2026-08-12 02:36:11 on dist-test-slave-2w3w"
server_key: "35b269d39a440a67a896716e119a0a48"
server_key_iv: "0f0a7a4dd421b8e2f19581ab9248bc09"
server_key_version: "encryptionkey@0"
I20260812 02:36:11.208926 1650 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260812 02:36:11.211576 1665 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:36:11.212879 1650 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260812 02:36:11.213037 1650 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501898713699-14748-0/minicluster-data/master-0/data,/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501898713699-14748-0/minicluster-data/master-0/wal
uuid: "185ba7148bd040a4b144023f244d5e1d"
format_stamp: "Formatted at 2026-08-12 02:36:11 on dist-test-slave-2w3w"
server_key: "35b269d39a440a67a896716e119a0a48"
server_key_iv: "0f0a7a4dd421b8e2f19581ab9248bc09"
server_key_version: "encryptionkey@0"
I20260812 02:36:11.213166 1650 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501898713699-14748-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501898713699-14748-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501898713699-14748-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:36:11.230142 1650 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:36:11.233450 1650 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:36:11.233675 1650 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:36:11.241710 1650 rpc_server.cc:307] RPC server started. Bound to: 127.14.103.62:44451
I20260812 02:36:11.241737 1723 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.103.62:44451 every 8 connection(s)
I20260812 02:36:11.242738 1650 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501898713699-14748-0/minicluster-data/master-0/data/info.pb
I20260812 02:36:11.243386 14748 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbep4M4/build/debug/bin/kudu as pid 1650
I20260812 02:36:11.243546 14748 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501898713699-14748-0/minicluster-data/master-0/wal/instance
I20260812 02:36:11.243860 14748 external_mini_cluster.cc:1428] Setting key 1f9843f9b06e204d82bc5b443bb02062
I20260812 02:36:11.246004 1726 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:36:11.251632 1726 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 185ba7148bd040a4b144023f244d5e1d: Bootstrap starting.
Aug 12 02:36:11 dist-test-slave-2w3w krb5kdc[1634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502171, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.103.62@KRBTEST.COM
I20260812 02:36:11.254065 1726 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 185ba7148bd040a4b144023f244d5e1d: Neither blocks nor log segments found. Creating new log.
I20260812 02:36:11.254858 1726 log.cc:824] T 00000000000000000000000000000000 P 185ba7148bd040a4b144023f244d5e1d: Log is configured to *not* fsync() on all Append() calls
I20260812 02:36:11.256816 1726 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 185ba7148bd040a4b144023f244d5e1d: No bootstrap required, opened a new log
I20260812 02:36:11.257933 1729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:11.245278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52442 (local address 127.14.103.62:44451)
0812 02:36:11.245754 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0812 02:36:11.245782 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0812 02:36:11.245818 (+ 36us) server_negotiation.cc:407] Connection header received
0812 02:36:11.246554 (+ 736us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:36:11.246572 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:36:11.246893 (+ 321us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:36:11.247289 (+ 396us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:36:11.248111 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:11.249126 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:11.250033 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:11.250504 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:11.250677 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:36:11.253263 (+ 2586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:36:11.253268 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:36:11.253279 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:36:11.253307 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:36:11.255197 (+ 1890us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:36:11.255659 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:36:11.255664 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:36:11.255668 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:36:11.255776 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:36:11.256064 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:36:11.256068 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:36:11.256070 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:36:11.256473 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:36:11.256688 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0812 02:36:11.256920 (+ 232us) server_negotiation.cc:299] Negotiation successful
0812 02:36:11.257224 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":161,"threads_started":1}
I20260812 02:36:11.260442 1726 raft_consensus.cc:348] T 00000000000000000000000000000000 P 185ba7148bd040a4b144023f244d5e1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "185ba7148bd040a4b144023f244d5e1d" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 44451 } }
I20260812 02:36:11.260752 1726 raft_consensus.cc:374] T 00000000000000000000000000000000 P 185ba7148bd040a4b144023f244d5e1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:36:11.260816 1726 raft_consensus.cc:729] T 00000000000000000000000000000000 P 185ba7148bd040a4b144023f244d5e1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 185ba7148bd040a4b144023f244d5e1d, State: Initialized, Role: FOLLOWER
I20260812 02:36:11.261273 1726 consensus_queue.cc:260] T 00000000000000000000000000000000 P 185ba7148bd040a4b144023f244d5e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "185ba7148bd040a4b144023f244d5e1d" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 44451 } }
I20260812 02:36:11.261402 1726 raft_consensus.cc:388] T 00000000000000000000000000000000 P 185ba7148bd040a4b144023f244d5e1d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 02:36:11.261452 1726 raft_consensus.cc:482] T 00000000000000000000000000000000 P 185ba7148bd040a4b144023f244d5e1d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 02:36:11.261523 1726 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 185ba7148bd040a4b144023f244d5e1d [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:36:11.262702 1726 raft_consensus.cc:504] T 00000000000000000000000000000000 P 185ba7148bd040a4b144023f244d5e1d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "185ba7148bd040a4b144023f244d5e1d" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 44451 } }
I20260812 02:36:11.263118 1726 leader_election.cc:302] T 00000000000000000000000000000000 P 185ba7148bd040a4b144023f244d5e1d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 185ba7148bd040a4b144023f244d5e1d; no voters:
I20260812 02:36:11.263406 1726 leader_election.cc:288] T 00000000000000000000000000000000 P 185ba7148bd040a4b144023f244d5e1d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 02:36:11.263562 1732 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 185ba7148bd040a4b144023f244d5e1d [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:36:11.263808 1732 raft_consensus.cc:686] T 00000000000000000000000000000000 P 185ba7148bd040a4b144023f244d5e1d [term 1 LEADER]: Becoming Leader. State: Replica: 185ba7148bd040a4b144023f244d5e1d, State: Running, Role: LEADER
I20260812 02:36:11.264103 1732 consensus_queue.cc:237] T 00000000000000000000000000000000 P 185ba7148bd040a4b144023f244d5e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "185ba7148bd040a4b144023f244d5e1d" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 44451 } }
I20260812 02:36:11.264657 1726 sys_catalog.cc:564] T 00000000000000000000000000000000 P 185ba7148bd040a4b144023f244d5e1d [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:36:11.265767 1733 sys_catalog.cc:455] T 00000000000000000000000000000000 P 185ba7148bd040a4b144023f244d5e1d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "185ba7148bd040a4b144023f244d5e1d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "185ba7148bd040a4b144023f244d5e1d" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 44451 } } }
I20260812 02:36:11.265906 1733 sys_catalog.cc:458] T 00000000000000000000000000000000 P 185ba7148bd040a4b144023f244d5e1d [sys.catalog]: This master's current role is: LEADER
I20260812 02:36:11.266207 1734 sys_catalog.cc:455] T 00000000000000000000000000000000 P 185ba7148bd040a4b144023f244d5e1d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 185ba7148bd040a4b144023f244d5e1d. Latest consensus state: current_term: 1 leader_uuid: "185ba7148bd040a4b144023f244d5e1d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "185ba7148bd040a4b144023f244d5e1d" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 44451 } } }
I20260812 02:36:11.266304 1734 sys_catalog.cc:458] T 00000000000000000000000000000000 P 185ba7148bd040a4b144023f244d5e1d [sys.catalog]: This master's current role is: LEADER
I20260812 02:36:11.266399 1741 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 02:36:11.269692 1741 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 02:36:11.276263 1741 catalog_manager.cc:1347] Generated new cluster ID: f33ab56b38374031bebc2a2524dc876e
I20260812 02:36:11.276355 1741 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 02:36:11.285729 1741 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 02:36:11.286811 1741 catalog_manager.cc:1504] Loading token signing keys...
I20260812 02:36:11.293690 1741 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 185ba7148bd040a4b144023f244d5e1d: Generated new TSK 0
I20260812 02:36:11.294445 1741 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.103.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.103.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.103.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.1.keytab.
Entry for principal kudu/127.14.103.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.1.keytab.
I20260812 02:36:11.336524 14748 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbep4M4/build/debug/bin/kudu
/tmp/dist-test-taskbep4M4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501898713699-14748-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501898713699-14748-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501898713699-14748-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501898713699-14748-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.103.1:0
--local_ip_for_outbound_sockets=127.14.103.1
--webserver_interface=127.14.103.1
--webserver_port=0
--tserver_master_addrs=127.14.103.62:44451
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.1.keytab
--principal=kudu/127.14.103.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.103.20:43843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501898713699-14748-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501898713699-14748-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501898713699-14748-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:36:11.480926 1755 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:36:11.481269 1755 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:36:11.481350 1755 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:36:11.485710 1755 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:36:11.485798 1755 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:36:11.485883 1755 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.103.1
I20260812 02:36:11.491459 1755 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.103.20:43843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501898713699-14748-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501898713699-14748-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.1.keytab
--principal=kudu/127.14.103.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.103.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501898713699-14748-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.103.1
--webserver_port=0
--tserver_master_addrs=127.14.103.62:44451
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1755
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.103.1
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501898713699-14748-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:36:11.492494 1755 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:36:11.493613 1755 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:36:11.503947 1762 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:36:11.503949 1761 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:36:11.504015 1755 server_base.cc:1034] running on GCE node
W20260812 02:36:11.503926 1764 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:36:11.504766 1755 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:36:11.505487 1755 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:36:11.506708 1755 hybrid_clock.cc:648] HybridClock initialized: now 1786502171506687 us; error 37 us; skew 500 ppm
Aug 12 02:36:11 dist-test-slave-2w3w krb5kdc[1634](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502171, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:36:11.509943 1755 init.cc:377] Logged in from keytab as kudu/127.14.103.1@KRBTEST.COM (short username kudu)
I20260812 02:36:11.510943 1755 webserver.cc:460] Webserver started at http://127.14.103.1:40881/ using document root <none> and password file <none>
I20260812 02:36:11.511622 1755 fs_manager.cc:359] Metadata directory not provided
I20260812 02:36:11.511686 1755 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:36:11.511906 1755 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:36:11.514076 1755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501898713699-14748-0/minicluster-data/ts-0/data/instance:
uuid: "ca6a34962ea2410696f5a0a12631d074"
format_stamp: "Formatted at 2026-08-12 02:36:11 on dist-test-slave-2w3w"
server_key: "ff103ee8d8ffff1bfab8c7f05a7ef5b4"
server_key_iv: "891e5fb03916a54c85797acc512ee188"
server_key_version: "encryptionkey@0"
I20260812 02:36:11.514671 1755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501898713699-14748-0/minicluster-data/ts-0/wal/instance:
uuid: "ca6a34962ea2410696f5a0a12631d074"
format_stamp: "Formatted at 2026-08-12 02:36:11 on dist-test-slave-2w3w"
server_key: "ff103ee8d8ffff1bfab8c7f05a7ef5b4"
server_key_iv: "891e5fb03916a54c85797acc512ee188"
server_key_version: "encryptionkey@0"
I20260812 02:36:11.518855 1755 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.000s
I20260812 02:36:11.521503 1772 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:36:11.522728 1755 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260812 02:36:11.522825 1755 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501898713699-14748-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501898713699-14748-0/minicluster-data/ts-0/wal
uuid: "ca6a34962ea2410696f5a0a12631d074"
format_stamp: "Formatted at 2026-08-12 02:36:11 on dist-test-slave-2w3w"
server_key: "ff103ee8d8ffff1bfab8c7f05a7ef5b4"
server_key_iv: "891e5fb03916a54c85797acc512ee188"
server_key_version: "encryptionkey@0"
I20260812 02:36:11.523010 1755 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501898713699-14748-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501898713699-14748-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501898713699-14748-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:36:11.574924 1755 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:36:11.578550 1755 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:36:11.578778 1755 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:36:11.579484 1755 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:36:11.580359 1755 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:36:11.580410 1755 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:36:11.580485 1755 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:36:11.580518 1755 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:36:11.589993 1755 rpc_server.cc:307] RPC server started. Bound to: 127.14.103.1:40853
I20260812 02:36:11.590022 1900 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.103.1:40853 every 8 connection(s)
I20260812 02:36:11.590946 1755 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501898713699-14748-0/minicluster-data/ts-0/data/info.pb
I20260812 02:36:11.596693 14748 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbep4M4/build/debug/bin/kudu as pid 1755
I20260812 02:36:11.596861 14748 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501898713699-14748-0/minicluster-data/ts-0/wal/instance
I20260812 02:36:11.597170 14748 external_mini_cluster.cc:1428] Setting key d53a14c2f2d5d531d092edda7054df9e
Aug 12 02:36:11 dist-test-slave-2w3w krb5kdc[1634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502171, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.1@KRBTEST.COM for kudu/127.14.103.62@KRBTEST.COM
I20260812 02:36:11.606501 1729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:11.592884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.1:34239 (local address 127.14.103.62:44451)
0812 02:36:11.593061 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0812 02:36:11.593066 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:36:11.593893 (+ 827us) server_negotiation.cc:407] Connection header received
0812 02:36:11.595245 (+ 1352us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:36:11.595248 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:36:11.595301 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:36:11.595399 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:36:11.596635 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:11.597331 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:11.598895 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:11.599253 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:11.599351 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:36:11.602586 (+ 3235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:36:11.602590 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:36:11.602592 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:36:11.602619 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:36:11.604140 (+ 1521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:36:11.604753 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:36:11.604757 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:36:11.604759 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:36:11.604856 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:36:11.605266 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:36:11.605270 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:36:11.605272 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:36:11.605490 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:36:11.605609 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0812 02:36:11.606169 (+ 560us) server_negotiation.cc:299] Negotiation successful
0812 02:36:11.606327 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260812 02:36:11.607615 1906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:11.593289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:44451 (local address 127.14.103.1:34239)
0812 02:36:11.593732 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0812 02:36:11.593777 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0812 02:36:11.595041 (+ 1264us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:36:11.595545 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:36:11.595551 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:36:11.595906 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:36:11.596459 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:36:11.596481 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:36:11.597480 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:36:11.597484 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:36:11.598719 (+ 1235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:36:11.598732 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:36:11.599435 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:36:11.599443 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:36:11.599688 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:36:11.600465 (+ 777us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:36:11.600491 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:36:11.602334 (+ 1843us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:36:11.604287 (+ 1953us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:36:11.604293 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:36:11.604312 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:36:11.604634 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:36:11.604984 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:36:11.604987 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:36:11.604989 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:36:11.605157 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:36:11.605609 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:36:11.605618 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:36:11.605913 (+ 295us) client_negotiation.cc:769] Sending connection context
0812 02:36:11.606125 (+ 212us) client_negotiation.cc:240] Negotiation successful
0812 02:36:11.606500 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":115,"threads_started":1}
WARNING: no policy specified for kudu/127.14.103.2@KRBTEST.COM; defaulting to no policy
I20260812 02:36:11.609113 1903 heartbeater.cc:347] Connected to a master server at 127.14.103.62:44451
I20260812 02:36:11.609447 1903 heartbeater.cc:464] Registering TS with master...
I20260812 02:36:11.609992 1903 heartbeater.cc:511] Master 127.14.103.62:44451 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.103.2@KRBTEST.COM" created.
I20260812 02:36:11.611608 1685 ts_manager.cc:194] Registered new tserver with Master: ca6a34962ea2410696f5a0a12631d074 (127.14.103.1:40853)
I20260812 02:36:11.612794 1685 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.103.1@KRBTEST.COM'} at 127.14.103.1:34239
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.103.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.2.keytab.
Entry for principal kudu/127.14.103.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.2.keytab.
I20260812 02:36:11.631336 14748 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbep4M4/build/debug/bin/kudu
/tmp/dist-test-taskbep4M4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501898713699-14748-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501898713699-14748-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501898713699-14748-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501898713699-14748-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.103.2:0
--local_ip_for_outbound_sockets=127.14.103.2
--webserver_interface=127.14.103.2
--webserver_port=0
--tserver_master_addrs=127.14.103.62:44451
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.2.keytab
--principal=kudu/127.14.103.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.103.20:43843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501898713699-14748-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501898713699-14748-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501898713699-14748-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:36:11.750542 1909 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:36:11.750850 1909 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:36:11.750903 1909 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:36:11.755367 1909 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:36:11.755456 1909 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:36:11.755556 1909 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.103.2
I20260812 02:36:11.761634 1909 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.103.20:43843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501898713699-14748-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501898713699-14748-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.2.keytab
--principal=kudu/127.14.103.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.103.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501898713699-14748-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.103.2
--webserver_port=0
--tserver_master_addrs=127.14.103.62:44451
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1909
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.103.2
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501898713699-14748-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:36:11.762811 1909 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:36:11.763914 1909 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:36:11.774467 1916 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:36:11.774590 1917 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:36:11.774863 1909 server_base.cc:1034] running on GCE node
W20260812 02:36:11.774451 1919 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:36:11.775487 1909 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:36:11.776283 1909 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:36:11.777514 1909 hybrid_clock.cc:648] HybridClock initialized: now 1786502171777495 us; error 32 us; skew 500 ppm
Aug 12 02:36:11 dist-test-slave-2w3w krb5kdc[1634](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502171, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:36:11.781095 1909 init.cc:377] Logged in from keytab as kudu/127.14.103.2@KRBTEST.COM (short username kudu)
I20260812 02:36:11.782205 1909 webserver.cc:460] Webserver started at http://127.14.103.2:33179/ using document root <none> and password file <none>
I20260812 02:36:11.782882 1909 fs_manager.cc:359] Metadata directory not provided
I20260812 02:36:11.783023 1909 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:36:11.783286 1909 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:36:11.785851 1909 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501898713699-14748-0/minicluster-data/ts-1/data/instance:
uuid: "68989c4070784506b8b90d7e93027b94"
format_stamp: "Formatted at 2026-08-12 02:36:11 on dist-test-slave-2w3w"
server_key: "19106d1692b019ce9aeadd4d09f660a2"
server_key_iv: "5ab8c9bbf05178589862cdc01e3174fb"
server_key_version: "encryptionkey@0"
I20260812 02:36:11.786557 1909 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501898713699-14748-0/minicluster-data/ts-1/wal/instance:
uuid: "68989c4070784506b8b90d7e93027b94"
format_stamp: "Formatted at 2026-08-12 02:36:11 on dist-test-slave-2w3w"
server_key: "19106d1692b019ce9aeadd4d09f660a2"
server_key_iv: "5ab8c9bbf05178589862cdc01e3174fb"
server_key_version: "encryptionkey@0"
I20260812 02:36:11.791100 1909 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.008s sys 0.000s
I20260812 02:36:11.794040 1926 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:36:11.795851 1909 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 02:36:11.796025 1909 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501898713699-14748-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501898713699-14748-0/minicluster-data/ts-1/wal
uuid: "68989c4070784506b8b90d7e93027b94"
format_stamp: "Formatted at 2026-08-12 02:36:11 on dist-test-slave-2w3w"
server_key: "19106d1692b019ce9aeadd4d09f660a2"
server_key_iv: "5ab8c9bbf05178589862cdc01e3174fb"
server_key_version: "encryptionkey@0"
I20260812 02:36:11.796169 1909 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501898713699-14748-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501898713699-14748-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501898713699-14748-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:36:11.814038 1909 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:36:11.817790 1909 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:36:11.818041 1909 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:36:11.818686 1909 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:36:11.819667 1909 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:36:11.819744 1909 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:36:11.819815 1909 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:36:11.819868 1909 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:36:11.828868 1909 rpc_server.cc:307] RPC server started. Bound to: 127.14.103.2:44039
I20260812 02:36:11.828933 2059 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.103.2:44039 every 8 connection(s)
I20260812 02:36:11.829850 1909 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501898713699-14748-0/minicluster-data/ts-1/data/info.pb
Aug 12 02:36:11 dist-test-slave-2w3w krb5kdc[1634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502171, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.2@KRBTEST.COM for kudu/127.14.103.62@KRBTEST.COM
I20260812 02:36:11.839704 14748 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbep4M4/build/debug/bin/kudu as pid 1909
I20260812 02:36:11.839825 14748 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501898713699-14748-0/minicluster-data/ts-1/wal/instance
I20260812 02:36:11.840154 14748 external_mini_cluster.cc:1428] Setting key 333a473cb89a33e4b0c0f76723dc4a88
I20260812 02:36:11.845057 1729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:11.831844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.2:47021 (local address 127.14.103.62:44451)
0812 02:36:11.831994 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0812 02:36:11.831998 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:36:11.832847 (+ 849us) server_negotiation.cc:407] Connection header received
0812 02:36:11.834211 (+ 1364us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:36:11.834214 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:36:11.834266 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:36:11.834342 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:36:11.835583 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:11.836135 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:11.837369 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:11.837592 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:11.837674 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:36:11.840513 (+ 2839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:36:11.840516 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:36:11.840518 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:36:11.840542 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:36:11.842245 (+ 1703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:36:11.843036 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:36:11.843040 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:36:11.843041 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:36:11.843133 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:36:11.843599 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:36:11.843603 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:36:11.843604 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:36:11.843803 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:36:11.843892 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0812 02:36:11.844652 (+ 760us) server_negotiation.cc:299] Negotiation successful
0812 02:36:11.844828 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260812 02:36:11.845829 2062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:11.832221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:44451 (local address 127.14.103.2:47021)
0812 02:36:11.832687 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0812 02:36:11.832731 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0812 02:36:11.833989 (+ 1258us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:36:11.834512 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:36:11.834518 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:36:11.834900 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:36:11.835457 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:36:11.835471 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:36:11.836279 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:36:11.836283 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:36:11.837248 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:36:11.837257 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:36:11.837728 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:36:11.837730 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:36:11.837900 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:36:11.838569 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:36:11.838585 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:36:11.840299 (+ 1714us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:36:11.842403 (+ 2104us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:36:11.842409 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:36:11.842429 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:36:11.842845 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:36:11.843235 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:36:11.843241 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:36:11.843246 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:36:11.843466 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:36:11.843920 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:36:11.843929 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:36:11.844310 (+ 381us) client_negotiation.cc:769] Sending connection context
0812 02:36:11.844548 (+ 238us) client_negotiation.cc:240] Negotiation successful
0812 02:36:11.844838 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":127,"threads_started":1}
I20260812 02:36:11.847321 2060 heartbeater.cc:347] Connected to a master server at 127.14.103.62:44451
I20260812 02:36:11.847606 2060 heartbeater.cc:464] Registering TS with master...
I20260812 02:36:11.848265 2060 heartbeater.cc:511] Master 127.14.103.62:44451 requested a full tablet report, sending...
I20260812 02:36:11.849560 1685 ts_manager.cc:194] Registered new tserver with Master: 68989c4070784506b8b90d7e93027b94 (127.14.103.2:44039)
I20260812 02:36:11.850438 1685 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.103.2@KRBTEST.COM'} at 127.14.103.2:47021
WARNING: no policy specified for kudu/127.14.103.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.103.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.103.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.3.keytab.
Entry for principal kudu/127.14.103.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.3.keytab.
I20260812 02:36:11.875598 14748 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbep4M4/build/debug/bin/kudu
/tmp/dist-test-taskbep4M4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501898713699-14748-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501898713699-14748-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501898713699-14748-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501898713699-14748-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.103.3:0
--local_ip_for_outbound_sockets=127.14.103.3
--webserver_interface=127.14.103.3
--webserver_port=0
--tserver_master_addrs=127.14.103.62:44451
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.3.keytab
--principal=kudu/127.14.103.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.103.20:43843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501898713699-14748-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501898713699-14748-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501898713699-14748-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:36:12.002130 2065 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:36:12.002530 2065 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:36:12.002616 2065 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:36:12.007059 2065 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:36:12.007167 2065 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:36:12.007266 2065 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.103.3
I20260812 02:36:12.013204 2065 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.103.20:43843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501898713699-14748-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501898713699-14748-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.3.keytab
--principal=kudu/127.14.103.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.103.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501898713699-14748-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.103.3
--webserver_port=0
--tserver_master_addrs=127.14.103.62:44451
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2065
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.103.3
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501898713699-14748-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:36:12.014490 2065 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:36:12.015761 2065 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:36:12.026836 2072 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:36:12.027084 2065 server_base.cc:1034] running on GCE node
W20260812 02:36:12.027155 2075 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:36:12.026836 2071 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:36:12.027742 2065 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:36:12.028438 2065 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:36:12.029630 2065 hybrid_clock.cc:648] HybridClock initialized: now 1786502172029605 us; error 50 us; skew 500 ppm
Aug 12 02:36:12 dist-test-slave-2w3w krb5kdc[1634](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502172, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:36:12.032812 2065 init.cc:377] Logged in from keytab as kudu/127.14.103.3@KRBTEST.COM (short username kudu)
I20260812 02:36:12.033826 2065 webserver.cc:460] Webserver started at http://127.14.103.3:43443/ using document root <none> and password file <none>
I20260812 02:36:12.034389 2065 fs_manager.cc:359] Metadata directory not provided
I20260812 02:36:12.034461 2065 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:36:12.034637 2065 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:36:12.037140 2065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501898713699-14748-0/minicluster-data/ts-2/data/instance:
uuid: "96513fb661454619a601f5d1a4ad60a2"
format_stamp: "Formatted at 2026-08-12 02:36:12 on dist-test-slave-2w3w"
server_key: "23772b487662a92f8e9f46a325c1a78f"
server_key_iv: "9f469438666f853f13fe2d9a367b606d"
server_key_version: "encryptionkey@0"
I20260812 02:36:12.037719 2065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501898713699-14748-0/minicluster-data/ts-2/wal/instance:
uuid: "96513fb661454619a601f5d1a4ad60a2"
format_stamp: "Formatted at 2026-08-12 02:36:12 on dist-test-slave-2w3w"
server_key: "23772b487662a92f8e9f46a325c1a78f"
server_key_iv: "9f469438666f853f13fe2d9a367b606d"
server_key_version: "encryptionkey@0"
I20260812 02:36:12.041908 2065 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260812 02:36:12.044756 2083 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:36:12.046144 2065 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260812 02:36:12.046263 2065 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501898713699-14748-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501898713699-14748-0/minicluster-data/ts-2/wal
uuid: "96513fb661454619a601f5d1a4ad60a2"
format_stamp: "Formatted at 2026-08-12 02:36:12 on dist-test-slave-2w3w"
server_key: "23772b487662a92f8e9f46a325c1a78f"
server_key_iv: "9f469438666f853f13fe2d9a367b606d"
server_key_version: "encryptionkey@0"
I20260812 02:36:12.046432 2065 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501898713699-14748-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501898713699-14748-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501898713699-14748-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:36:12.061179 2065 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:36:12.065268 2065 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:36:12.065514 2065 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:36:12.066330 2065 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:36:12.067344 2065 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:36:12.067400 2065 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:36:12.067476 2065 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:36:12.067514 2065 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:36:12.078022 2065 rpc_server.cc:307] RPC server started. Bound to: 127.14.103.3:39189
I20260812 02:36:12.078048 2210 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.103.3:39189 every 8 connection(s)
I20260812 02:36:12.079128 2065 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501898713699-14748-0/minicluster-data/ts-2/data/info.pb
I20260812 02:36:12.084618 14748 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbep4M4/build/debug/bin/kudu as pid 2065
I20260812 02:36:12.084851 14748 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501898713699-14748-0/minicluster-data/ts-2/wal/instance
I20260812 02:36:12.085309 14748 external_mini_cluster.cc:1428] Setting key 095d01625c488305a4b56c890feb8da5
Aug 12 02:36:12 dist-test-slave-2w3w krb5kdc[1634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502172, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.3@KRBTEST.COM for kudu/127.14.103.62@KRBTEST.COM
I20260812 02:36:12.096685 1729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:12.081272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.3:49387 (local address 127.14.103.62:44451)
0812 02:36:12.081466 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0812 02:36:12.081470 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:36:12.082358 (+ 888us) server_negotiation.cc:407] Connection header received
0812 02:36:12.084217 (+ 1859us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:36:12.084221 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:36:12.084320 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:36:12.084446 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:36:12.086150 (+ 1704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:12.087054 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:12.088478 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:12.088723 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:12.088817 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:36:12.092072 (+ 3255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:36:12.092076 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:36:12.092079 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:36:12.092136 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:36:12.093924 (+ 1788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:36:12.094755 (+ 831us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:36:12.094759 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:36:12.094760 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:36:12.094860 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:36:12.095368 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:36:12.095371 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:36:12.095373 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:36:12.095611 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:36:12.095710 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0812 02:36:12.096398 (+ 688us) server_negotiation.cc:299] Negotiation successful
0812 02:36:12.096536 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260812 02:36:12.097572 2213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:12.081635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:44451 (local address 127.14.103.3:49387)
0812 02:36:12.082170 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0812 02:36:12.082221 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0812 02:36:12.083932 (+ 1711us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:36:12.084644 (+ 712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:36:12.084653 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:36:12.085181 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:36:12.085978 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:36:12.085998 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:36:12.087224 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:36:12.087227 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:36:12.088327 (+ 1100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:36:12.088338 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:36:12.088883 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:36:12.088886 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:36:12.089043 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:36:12.089822 (+ 779us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:36:12.089840 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:36:12.091786 (+ 1946us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:36:12.094161 (+ 2375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:36:12.094168 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:36:12.094188 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:36:12.094586 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:36:12.095051 (+ 465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:36:12.095053 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:36:12.095055 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:36:12.095231 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:36:12.095744 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:36:12.095752 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:36:12.096076 (+ 324us) client_negotiation.cc:769] Sending connection context
0812 02:36:12.096264 (+ 188us) client_negotiation.cc:240] Negotiation successful
0812 02:36:12.096575 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":127,"threads_started":1}
I20260812 02:36:12.099257 2211 heartbeater.cc:347] Connected to a master server at 127.14.103.62:44451
I20260812 02:36:12.099586 2211 heartbeater.cc:464] Registering TS with master...
I20260812 02:36:12.100287 2211 heartbeater.cc:511] Master 127.14.103.62:44451 requested a full tablet report, sending...
I20260812 02:36:12.101538 1683 ts_manager.cc:194] Registered new tserver with Master: 96513fb661454619a601f5d1a4ad60a2 (127.14.103.3:39189)
I20260812 02:36:12.102176 1683 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.103.3@KRBTEST.COM'} at 127.14.103.3:49387
I20260812 02:36:12.112586 14748 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 02:36:12.123072 1729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:12.114778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52454 (local address 127.14.103.62:44451)
0812 02:36:12.114923 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0812 02:36:12.114927 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:36:12.115087 (+ 160us) server_negotiation.cc:407] Connection header received
0812 02:36:12.115157 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:36:12.115159 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:36:12.115205 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:36:12.115281 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:36:12.116013 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:12.116642 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:12.117529 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:12.117766 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:12.117858 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:36:12.119402 (+ 1544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:36:12.119405 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:36:12.119407 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:36:12.119435 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:36:12.121181 (+ 1746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:36:12.121692 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:36:12.121695 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:36:12.121697 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:36:12.121786 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:36:12.122093 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:36:12.122096 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:36:12.122097 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:36:12.122357 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:36:12.122474 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0812 02:36:12.122731 (+ 257us) server_negotiation.cc:299] Negotiation successful
0812 02:36:12.122840 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260812 02:36:12.126699 1683 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52454:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 02:36:12.130002 1683 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 02:36:12.145653 2229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:12.139807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38040 (local address 127.14.103.2:44039)
0812 02:36:12.140550 (+ 743us) server_negotiation.cc:206] Beginning negotiation
0812 02:36:12.140562 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 02:36:12.140584 (+ 22us) server_negotiation.cc:407] Connection header received
0812 02:36:12.140665 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:36:12.140671 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:36:12.140878 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:36:12.141032 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:36:12.141568 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:12.142903 (+ 1335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:12.144330 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:12.144890 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:12.145020 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:36:12.145187 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0812 02:36:12.145384 (+ 197us) server_negotiation.cc:299] Negotiation successful
0812 02:36:12.145486 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":643,"thread_start_us":116,"threads_started":1}
I20260812 02:36:12.145648 2223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:12.139105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.1:40853 (local address 127.0.0.1:36604)
0812 02:36:12.140064 (+ 959us) negotiation.cc:107] Waiting for socket to connect
0812 02:36:12.140134 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0812 02:36:12.140269 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:36:12.141069 (+ 800us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:36:12.141075 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:36:12.141100 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:36:12.141244 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:36:12.141252 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:36:12.142509 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:36:12.142515 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:36:12.143871 (+ 1356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:36:12.143892 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:36:12.145036 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:36:12.145040 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:36:12.145179 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:36:12.145221 (+ 42us) client_negotiation.cc:769] Sending connection context
0812 02:36:12.145311 (+ 90us) client_negotiation.cc:240] Negotiation successful
0812 02:36:12.145446 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":768,"mutex_wait_us":4,"thread_start_us":107,"threads_started":1}
I20260812 02:36:12.145646 2225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:12.139167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.2:44039 (local address 127.0.0.1:38040)
0812 02:36:12.140157 (+ 990us) negotiation.cc:107] Waiting for socket to connect
0812 02:36:12.140172 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 02:36:12.140279 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:36:12.141218 (+ 939us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:36:12.141226 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:36:12.141248 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:36:12.141436 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:36:12.141445 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:36:12.143100 (+ 1655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:36:12.143103 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:36:12.144210 (+ 1107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:36:12.144218 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:36:12.145020 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:36:12.145024 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:36:12.145146 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:36:12.145239 (+ 93us) client_negotiation.cc:769] Sending connection context
0812 02:36:12.145348 (+ 109us) client_negotiation.cc:240] Negotiation successful
0812 02:36:12.145448 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":896,"mutex_wait_us":178,"spinlock_wait_cycles":2560,"thread_start_us":114,"threads_started":1}
I20260812 02:36:12.146162 2228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:12.139436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36604 (local address 127.14.103.1:40853)
0812 02:36:12.140551 (+ 1115us) server_negotiation.cc:206] Beginning negotiation
0812 02:36:12.140560 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 02:36:12.140602 (+ 42us) server_negotiation.cc:407] Connection header received
0812 02:36:12.140670 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:36:12.140674 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:36:12.140878 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:36:12.141054 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:36:12.141404 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:12.142364 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:12.144024 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:12.144881 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:12.145719 (+ 838us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:36:12.145847 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0812 02:36:12.145896 (+ 49us) server_negotiation.cc:299] Negotiation successful
0812 02:36:12.146007 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1006,"thread_start_us":99,"threads_started":1}
I20260812 02:36:12.146262 2224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:12.139445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.3:39189 (local address 127.0.0.1:40014)
0812 02:36:12.140157 (+ 712us) negotiation.cc:107] Waiting for socket to connect
0812 02:36:12.140172 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 02:36:12.140279 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:36:12.141220 (+ 941us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:36:12.141223 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:36:12.141236 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:36:12.141377 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:36:12.141383 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:36:12.143280 (+ 1897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:36:12.143284 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:36:12.144563 (+ 1279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:36:12.144575 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:36:12.145851 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:36:12.145855 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:36:12.145958 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:36:12.145980 (+ 22us) client_negotiation.cc:769] Sending connection context
0812 02:36:12.146052 (+ 72us) client_negotiation.cc:240] Negotiation successful
0812 02:36:12.146113 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":666,"mutex_wait_us":31,"spinlock_wait_cycles":7808,"thread_start_us":99,"threads_started":1}
I20260812 02:36:12.147084 2231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:12.140012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40014 (local address 127.14.103.3:39189)
0812 02:36:12.140563 (+ 551us) server_negotiation.cc:206] Beginning negotiation
0812 02:36:12.140576 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0812 02:36:12.140599 (+ 23us) server_negotiation.cc:407] Connection header received
0812 02:36:12.140671 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:36:12.140676 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:36:12.140900 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:36:12.141052 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:36:12.141498 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:12.142903 (+ 1405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:12.144713 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:12.145546 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:12.145647 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:36:12.145787 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0812 02:36:12.146791 (+ 1004us) server_negotiation.cc:299] Negotiation successful
0812 02:36:12.146892 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":454,"thread_start_us":131,"threads_started":1}
I20260812 02:36:12.148798 1982 tablet_service.cc:1467] Processing CreateTablet for tablet b59b6f544f4d453a9257fcff87a8c5af (DEFAULT_TABLE table=test-table [id=27e16a04e8654d74b8cbd5b98a12eef8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:36:12.149138 1825 tablet_service.cc:1467] Processing CreateTablet for tablet b59b6f544f4d453a9257fcff87a8c5af (DEFAULT_TABLE table=test-table [id=27e16a04e8654d74b8cbd5b98a12eef8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:36:12.149978 1982 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b59b6f544f4d453a9257fcff87a8c5af. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:36:12.149680 2134 tablet_service.cc:1467] Processing CreateTablet for tablet b59b6f544f4d453a9257fcff87a8c5af (DEFAULT_TABLE table=test-table [id=27e16a04e8654d74b8cbd5b98a12eef8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:36:12.150189 1825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b59b6f544f4d453a9257fcff87a8c5af. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:36:12.150660 2134 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b59b6f544f4d453a9257fcff87a8c5af. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:36:12.157115 2232 tablet_bootstrap.cc:492] T b59b6f544f4d453a9257fcff87a8c5af P 68989c4070784506b8b90d7e93027b94: Bootstrap starting.
I20260812 02:36:12.158522 2233 tablet_bootstrap.cc:492] T b59b6f544f4d453a9257fcff87a8c5af P ca6a34962ea2410696f5a0a12631d074: Bootstrap starting.
I20260812 02:36:12.158522 2234 tablet_bootstrap.cc:492] T b59b6f544f4d453a9257fcff87a8c5af P 96513fb661454619a601f5d1a4ad60a2: Bootstrap starting.
I20260812 02:36:12.160075 2232 tablet_bootstrap.cc:654] T b59b6f544f4d453a9257fcff87a8c5af P 68989c4070784506b8b90d7e93027b94: Neither blocks nor log segments found. Creating new log.
I20260812 02:36:12.160754 2233 tablet_bootstrap.cc:654] T b59b6f544f4d453a9257fcff87a8c5af P ca6a34962ea2410696f5a0a12631d074: Neither blocks nor log segments found. Creating new log.
I20260812 02:36:12.160760 2234 tablet_bootstrap.cc:654] T b59b6f544f4d453a9257fcff87a8c5af P 96513fb661454619a601f5d1a4ad60a2: Neither blocks nor log segments found. Creating new log.
I20260812 02:36:12.161180 2232 log.cc:824] T b59b6f544f4d453a9257fcff87a8c5af P 68989c4070784506b8b90d7e93027b94: Log is configured to *not* fsync() on all Append() calls
I20260812 02:36:12.161753 2233 log.cc:824] T b59b6f544f4d453a9257fcff87a8c5af P ca6a34962ea2410696f5a0a12631d074: Log is configured to *not* fsync() on all Append() calls
I20260812 02:36:12.162070 2234 log.cc:824] T b59b6f544f4d453a9257fcff87a8c5af P 96513fb661454619a601f5d1a4ad60a2: Log is configured to *not* fsync() on all Append() calls
I20260812 02:36:12.163383 2232 tablet_bootstrap.cc:492] T b59b6f544f4d453a9257fcff87a8c5af P 68989c4070784506b8b90d7e93027b94: No bootstrap required, opened a new log
I20260812 02:36:12.163602 2232 ts_tablet_manager.cc:1397] T b59b6f544f4d453a9257fcff87a8c5af P 68989c4070784506b8b90d7e93027b94: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260812 02:36:12.163638 2233 tablet_bootstrap.cc:492] T b59b6f544f4d453a9257fcff87a8c5af P ca6a34962ea2410696f5a0a12631d074: No bootstrap required, opened a new log
I20260812 02:36:12.163841 2233 ts_tablet_manager.cc:1397] T b59b6f544f4d453a9257fcff87a8c5af P ca6a34962ea2410696f5a0a12631d074: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260812 02:36:12.167443 2232 raft_consensus.cc:348] T b59b6f544f4d453a9257fcff87a8c5af P 68989c4070784506b8b90d7e93027b94 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca6a34962ea2410696f5a0a12631d074" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 40853 } } peers { permanent_uuid: "68989c4070784506b8b90d7e93027b94" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 44039 } } peers { permanent_uuid: "96513fb661454619a601f5d1a4ad60a2" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 39189 } }
I20260812 02:36:12.167762 2232 raft_consensus.cc:374] T b59b6f544f4d453a9257fcff87a8c5af P 68989c4070784506b8b90d7e93027b94 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:36:12.167851 2232 raft_consensus.cc:729] T b59b6f544f4d453a9257fcff87a8c5af P 68989c4070784506b8b90d7e93027b94 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68989c4070784506b8b90d7e93027b94, State: Initialized, Role: FOLLOWER
I20260812 02:36:12.167833 2233 raft_consensus.cc:348] T b59b6f544f4d453a9257fcff87a8c5af P ca6a34962ea2410696f5a0a12631d074 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca6a34962ea2410696f5a0a12631d074" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 40853 } } peers { permanent_uuid: "68989c4070784506b8b90d7e93027b94" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 44039 } } peers { permanent_uuid: "96513fb661454619a601f5d1a4ad60a2" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 39189 } }
I20260812 02:36:12.168095 2233 raft_consensus.cc:374] T b59b6f544f4d453a9257fcff87a8c5af P ca6a34962ea2410696f5a0a12631d074 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:36:12.168182 2233 raft_consensus.cc:729] T b59b6f544f4d453a9257fcff87a8c5af P ca6a34962ea2410696f5a0a12631d074 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca6a34962ea2410696f5a0a12631d074, State: Initialized, Role: FOLLOWER
I20260812 02:36:12.168385 2232 consensus_queue.cc:260] T b59b6f544f4d453a9257fcff87a8c5af P 68989c4070784506b8b90d7e93027b94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca6a34962ea2410696f5a0a12631d074" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 40853 } } peers { permanent_uuid: "68989c4070784506b8b90d7e93027b94" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 44039 } } peers { permanent_uuid: "96513fb661454619a601f5d1a4ad60a2" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 39189 } }
I20260812 02:36:12.168640 2233 consensus_queue.cc:260] T b59b6f544f4d453a9257fcff87a8c5af P ca6a34962ea2410696f5a0a12631d074 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca6a34962ea2410696f5a0a12631d074" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 40853 } } peers { permanent_uuid: "68989c4070784506b8b90d7e93027b94" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 44039 } } peers { permanent_uuid: "96513fb661454619a601f5d1a4ad60a2" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 39189 } }
I20260812 02:36:12.169183 2060 heartbeater.cc:503] Master 127.14.103.62:44451 was elected leader, sending a full tablet report...
I20260812 02:36:12.169402 1903 heartbeater.cc:503] Master 127.14.103.62:44451 was elected leader, sending a full tablet report...
I20260812 02:36:12.169631 2232 ts_tablet_manager.cc:1428] T b59b6f544f4d453a9257fcff87a8c5af P 68989c4070784506b8b90d7e93027b94: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260812 02:36:12.169673 2233 ts_tablet_manager.cc:1428] T b59b6f544f4d453a9257fcff87a8c5af P ca6a34962ea2410696f5a0a12631d074: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260812 02:36:12.171958 2234 tablet_bootstrap.cc:492] T b59b6f544f4d453a9257fcff87a8c5af P 96513fb661454619a601f5d1a4ad60a2: No bootstrap required, opened a new log
I20260812 02:36:12.172212 2234 ts_tablet_manager.cc:1397] T b59b6f544f4d453a9257fcff87a8c5af P 96513fb661454619a601f5d1a4ad60a2: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.000s
I20260812 02:36:12.176537 2234 raft_consensus.cc:348] T b59b6f544f4d453a9257fcff87a8c5af P 96513fb661454619a601f5d1a4ad60a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca6a34962ea2410696f5a0a12631d074" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 40853 } } peers { permanent_uuid: "68989c4070784506b8b90d7e93027b94" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 44039 } } peers { permanent_uuid: "96513fb661454619a601f5d1a4ad60a2" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 39189 } }
I20260812 02:36:12.176837 2234 raft_consensus.cc:374] T b59b6f544f4d453a9257fcff87a8c5af P 96513fb661454619a601f5d1a4ad60a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:36:12.176877 2234 raft_consensus.cc:729] T b59b6f544f4d453a9257fcff87a8c5af P 96513fb661454619a601f5d1a4ad60a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96513fb661454619a601f5d1a4ad60a2, State: Initialized, Role: FOLLOWER
I20260812 02:36:12.177361 2234 consensus_queue.cc:260] T b59b6f544f4d453a9257fcff87a8c5af P 96513fb661454619a601f5d1a4ad60a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca6a34962ea2410696f5a0a12631d074" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 40853 } } peers { permanent_uuid: "68989c4070784506b8b90d7e93027b94" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 44039 } } peers { permanent_uuid: "96513fb661454619a601f5d1a4ad60a2" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 39189 } }
I20260812 02:36:12.178157 2211 heartbeater.cc:503] Master 127.14.103.62:44451 was elected leader, sending a full tablet report...
I20260812 02:36:12.178493 2234 ts_tablet_manager.cc:1428] T b59b6f544f4d453a9257fcff87a8c5af P 96513fb661454619a601f5d1a4ad60a2: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
W20260812 02:36:12.330216 2212 tablet.cc:2369] T b59b6f544f4d453a9257fcff87a8c5af P 96513fb661454619a601f5d1a4ad60a2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 02:36:12.331033 2061 tablet.cc:2369] T b59b6f544f4d453a9257fcff87a8c5af P 68989c4070784506b8b90d7e93027b94: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 02:36:12.342444 1904 tablet.cc:2369] T b59b6f544f4d453a9257fcff87a8c5af P ca6a34962ea2410696f5a0a12631d074: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 02:36:12.499534 2241 raft_consensus.cc:482] T b59b6f544f4d453a9257fcff87a8c5af P 96513fb661454619a601f5d1a4ad60a2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 02:36:12.499751 2241 raft_consensus.cc:504] T b59b6f544f4d453a9257fcff87a8c5af P 96513fb661454619a601f5d1a4ad60a2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca6a34962ea2410696f5a0a12631d074" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 40853 } } peers { permanent_uuid: "68989c4070784506b8b90d7e93027b94" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 44039 } } peers { permanent_uuid: "96513fb661454619a601f5d1a4ad60a2" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 39189 } }
I20260812 02:36:12.501173 2241 leader_election.cc:288] T b59b6f544f4d453a9257fcff87a8c5af P 96513fb661454619a601f5d1a4ad60a2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ca6a34962ea2410696f5a0a12631d074 (127.14.103.1:40853), 68989c4070784506b8b90d7e93027b94 (127.14.103.2:44039)
I20260812 02:36:12.505584 2213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:12.501430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.1:40853 (local address 127.14.103.3:51285)
0812 02:36:12.501598 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0812 02:36:12.501617 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 02:36:12.501720 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:36:12.502202 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:36:12.502205 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:36:12.502227 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:36:12.502378 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:36:12.502385 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:36:12.503353 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:36:12.503356 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:36:12.504309 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:36:12.504319 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:36:12.505190 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:36:12.505193 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:36:12.505282 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:36:12.505300 (+ 18us) client_negotiation.cc:769] Sending connection context
0812 02:36:12.505364 (+ 64us) client_negotiation.cc:240] Negotiation successful
0812 02:36:12.505428 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68,"mutex_wait_us":39}
I20260812 02:36:12.505843 2228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:12.501672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.3:51285 (local address 127.14.103.1:40853)
0812 02:36:12.501883 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0812 02:36:12.501888 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:36:12.501907 (+ 19us) server_negotiation.cc:407] Connection header received
0812 02:36:12.502005 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:36:12.502011 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:36:12.502070 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:36:12.502152 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:36:12.502521 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:12.503204 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:12.504449 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:12.504962 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:12.505092 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:36:12.505134 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0812 02:36:12.505498 (+ 364us) server_negotiation.cc:299] Negotiation successful
0812 02:36:12.505607 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109}
I20260812 02:36:12.506510 2229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:12.501708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.3:52933 (local address 127.14.103.2:44039)
0812 02:36:12.501871 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0812 02:36:12.501876 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:36:12.502102 (+ 226us) server_negotiation.cc:407] Connection header received
0812 02:36:12.502202 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:36:12.502205 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:36:12.502257 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:36:12.502353 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:36:12.502665 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:12.503181 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:12.504460 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:12.504857 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:12.505005 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:36:12.505064 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0812 02:36:12.505964 (+ 900us) server_negotiation.cc:299] Negotiation successful
0812 02:36:12.506070 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260812 02:36:12.506898 2243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:12.501500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.2:44039 (local address 127.14.103.3:52933)
0812 02:36:12.501948 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0812 02:36:12.501962 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 02:36:12.502088 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:36:12.502370 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:36:12.502372 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:36:12.502386 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:36:12.502539 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:36:12.502547 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:36:12.503292 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:36:12.503299 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:36:12.504311 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:36:12.504321 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:36:12.505003 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:36:12.505007 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:36:12.505689 (+ 682us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:36:12.505800 (+ 111us) client_negotiation.cc:769] Sending connection context
0812 02:36:12.506703 (+ 903us) client_negotiation.cc:240] Negotiation successful
0812 02:36:12.506785 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":381,"mutex_wait_us":85,"thread_start_us":130,"threads_started":1}
I20260812 02:36:12.507017 1848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b59b6f544f4d453a9257fcff87a8c5af" candidate_uuid: "96513fb661454619a601f5d1a4ad60a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca6a34962ea2410696f5a0a12631d074" is_pre_election: true
I20260812 02:36:12.507439 1848 raft_consensus.cc:2454] T b59b6f544f4d453a9257fcff87a8c5af P ca6a34962ea2410696f5a0a12631d074 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 96513fb661454619a601f5d1a4ad60a2 in term 0.
I20260812 02:36:12.508148 2087 leader_election.cc:302] T b59b6f544f4d453a9257fcff87a8c5af P 96513fb661454619a601f5d1a4ad60a2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 96513fb661454619a601f5d1a4ad60a2, ca6a34962ea2410696f5a0a12631d074; no voters:
I20260812 02:36:12.508149 2005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b59b6f544f4d453a9257fcff87a8c5af" candidate_uuid: "96513fb661454619a601f5d1a4ad60a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68989c4070784506b8b90d7e93027b94" is_pre_election: true
I20260812 02:36:12.508468 2005 raft_consensus.cc:2454] T b59b6f544f4d453a9257fcff87a8c5af P 68989c4070784506b8b90d7e93027b94 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 96513fb661454619a601f5d1a4ad60a2 in term 0.
I20260812 02:36:12.508481 2241 raft_consensus.cc:2781] T b59b6f544f4d453a9257fcff87a8c5af P 96513fb661454619a601f5d1a4ad60a2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 02:36:12.508582 2241 raft_consensus.cc:482] T b59b6f544f4d453a9257fcff87a8c5af P 96513fb661454619a601f5d1a4ad60a2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 02:36:12.508630 2241 raft_consensus.cc:3037] T b59b6f544f4d453a9257fcff87a8c5af P 96513fb661454619a601f5d1a4ad60a2 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:36:12.509859 2241 raft_consensus.cc:504] T b59b6f544f4d453a9257fcff87a8c5af P 96513fb661454619a601f5d1a4ad60a2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca6a34962ea2410696f5a0a12631d074" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 40853 } } peers { permanent_uuid: "68989c4070784506b8b90d7e93027b94" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 44039 } } peers { permanent_uuid: "96513fb661454619a601f5d1a4ad60a2" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 39189 } }
I20260812 02:36:12.510212 2241 leader_election.cc:288] T b59b6f544f4d453a9257fcff87a8c5af P 96513fb661454619a601f5d1a4ad60a2 [CANDIDATE]: Term 1 election: Requested vote from peers ca6a34962ea2410696f5a0a12631d074 (127.14.103.1:40853), 68989c4070784506b8b90d7e93027b94 (127.14.103.2:44039)
I20260812 02:36:12.510638 2005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b59b6f544f4d453a9257fcff87a8c5af" candidate_uuid: "96513fb661454619a601f5d1a4ad60a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68989c4070784506b8b90d7e93027b94"
I20260812 02:36:12.510660 1848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b59b6f544f4d453a9257fcff87a8c5af" candidate_uuid: "96513fb661454619a601f5d1a4ad60a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca6a34962ea2410696f5a0a12631d074"
I20260812 02:36:12.510784 2005 raft_consensus.cc:3037] T b59b6f544f4d453a9257fcff87a8c5af P 68989c4070784506b8b90d7e93027b94 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:36:12.510808 1848 raft_consensus.cc:3037] T b59b6f544f4d453a9257fcff87a8c5af P ca6a34962ea2410696f5a0a12631d074 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:36:12.511935 2005 raft_consensus.cc:2454] T b59b6f544f4d453a9257fcff87a8c5af P 68989c4070784506b8b90d7e93027b94 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96513fb661454619a601f5d1a4ad60a2 in term 1.
I20260812 02:36:12.512033 1848 raft_consensus.cc:2454] T b59b6f544f4d453a9257fcff87a8c5af P ca6a34962ea2410696f5a0a12631d074 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96513fb661454619a601f5d1a4ad60a2 in term 1.
I20260812 02:36:12.512372 2084 leader_election.cc:302] T b59b6f544f4d453a9257fcff87a8c5af P 96513fb661454619a601f5d1a4ad60a2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 68989c4070784506b8b90d7e93027b94, 96513fb661454619a601f5d1a4ad60a2; no voters:
I20260812 02:36:12.512539 2241 raft_consensus.cc:2781] T b59b6f544f4d453a9257fcff87a8c5af P 96513fb661454619a601f5d1a4ad60a2 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:36:12.512781 2241 raft_consensus.cc:686] T b59b6f544f4d453a9257fcff87a8c5af P 96513fb661454619a601f5d1a4ad60a2 [term 1 LEADER]: Becoming Leader. State: Replica: 96513fb661454619a601f5d1a4ad60a2, State: Running, Role: LEADER
I20260812 02:36:12.513221 2241 consensus_queue.cc:237] T b59b6f544f4d453a9257fcff87a8c5af P 96513fb661454619a601f5d1a4ad60a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca6a34962ea2410696f5a0a12631d074" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 40853 } } peers { permanent_uuid: "68989c4070784506b8b90d7e93027b94" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 44039 } } peers { permanent_uuid: "96513fb661454619a601f5d1a4ad60a2" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 39189 } }
I20260812 02:36:12.516675 1683 catalog_manager.cc:5495] T b59b6f544f4d453a9257fcff87a8c5af P 96513fb661454619a601f5d1a4ad60a2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 96513fb661454619a601f5d1a4ad60a2 (127.14.103.3). New cstate: current_term: 1 leader_uuid: "96513fb661454619a601f5d1a4ad60a2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca6a34962ea2410696f5a0a12631d074" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 40853 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "68989c4070784506b8b90d7e93027b94" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 44039 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "96513fb661454619a601f5d1a4ad60a2" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 39189 } health_report { overall_health: HEALTHY } } }
I20260812 02:36:12.603549 2231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:12.599478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40018 (local address 127.14.103.3:39189)
0812 02:36:12.599640 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0812 02:36:12.599644 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:36:12.599659 (+ 15us) server_negotiation.cc:407] Connection header received
0812 02:36:12.599829 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:36:12.599832 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:36:12.599888 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:36:12.599977 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 02:36:12.600285 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:12.600898 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:12.601944 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:12.602184 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:12.602280 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:36:12.602570 (+ 290us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 02:36:12.603095 (+ 525us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 02:36:12.603189 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0812 02:36:12.603259 (+ 70us) server_negotiation.cc:299] Negotiation successful
0812 02:36:12.603314 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260812 02:36:12.609546 1848 raft_consensus.cc:1261] T b59b6f544f4d453a9257fcff87a8c5af P ca6a34962ea2410696f5a0a12631d074 [term 1 FOLLOWER]: Refusing update from remote peer 96513fb661454619a601f5d1a4ad60a2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:36:12.609546 2005 raft_consensus.cc:1261] T b59b6f544f4d453a9257fcff87a8c5af P 68989c4070784506b8b90d7e93027b94 [term 1 FOLLOWER]: Refusing update from remote peer 96513fb661454619a601f5d1a4ad60a2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:36:12.610293 2244 consensus_queue.cc:1035] T b59b6f544f4d453a9257fcff87a8c5af P 96513fb661454619a601f5d1a4ad60a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68989c4070784506b8b90d7e93027b94" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 44039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:36:12.610546 2241 consensus_queue.cc:1035] T b59b6f544f4d453a9257fcff87a8c5af P 96513fb661454619a601f5d1a4ad60a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca6a34962ea2410696f5a0a12631d074" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 40853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:36:12.620337 2248 mvcc.cc:205] Tried to move back new op lower bound from 7317512899000401920 to 7317512898616852480. Current Snapshot: MvccSnapshot[applied={T|T < 7317512899000401920}]
I20260812 02:36:12.626382 1683 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52454:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 02:36:12.626580 1683 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52454:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 02:36:12.629201 1683 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 185ba7148bd040a4b144023f244d5e1d: Sending DeleteTablet for 3 replicas of tablet b59b6f544f4d453a9257fcff87a8c5af
I20260812 02:36:12.630035 1825 tablet_service.cc:1514] Processing DeleteTablet for tablet b59b6f544f4d453a9257fcff87a8c5af with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:36:12 UTC) from {username='kudu', principal='kudu/127.14.103.62@KRBTEST.COM'} at 127.0.0.1:36604
I20260812 02:36:12.630089 1982 tablet_service.cc:1514] Processing DeleteTablet for tablet b59b6f544f4d453a9257fcff87a8c5af with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:36:12 UTC) from {username='kudu', principal='kudu/127.14.103.62@KRBTEST.COM'} at 127.0.0.1:38040
I20260812 02:36:12.630309 2134 tablet_service.cc:1514] Processing DeleteTablet for tablet b59b6f544f4d453a9257fcff87a8c5af with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:36:12 UTC) from {username='kudu', principal='kudu/127.14.103.62@KRBTEST.COM'} at 127.0.0.1:40014
I20260812 02:36:12.630746 2260 tablet_replica.cc:330] T b59b6f544f4d453a9257fcff87a8c5af P 96513fb661454619a601f5d1a4ad60a2: stopping tablet replica
I20260812 02:36:12.630887 2259 tablet_replica.cc:330] T b59b6f544f4d453a9257fcff87a8c5af P ca6a34962ea2410696f5a0a12631d074: stopping tablet replica
I20260812 02:36:12.631143 2260 raft_consensus.cc:2229] T b59b6f544f4d453a9257fcff87a8c5af P 96513fb661454619a601f5d1a4ad60a2 [term 1 LEADER]: Raft consensus shutting down.
I20260812 02:36:12.631192 2259 raft_consensus.cc:2229] T b59b6f544f4d453a9257fcff87a8c5af P ca6a34962ea2410696f5a0a12631d074 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 02:36:12.631613 2260 raft_consensus.cc:2258] T b59b6f544f4d453a9257fcff87a8c5af P 96513fb661454619a601f5d1a4ad60a2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:36:12.631677 14748 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbep4M4/build/debug/bin/kudu with pid 1755
I20260812 02:36:12.636555 2260 ts_tablet_manager.cc:1905] T b59b6f544f4d453a9257fcff87a8c5af P 96513fb661454619a601f5d1a4ad60a2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:36:12.640228 2260 ts_tablet_manager.cc:1918] T b59b6f544f4d453a9257fcff87a8c5af P 96513fb661454619a601f5d1a4ad60a2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 02:36:12.640401 2260 log.cc:1196] T b59b6f544f4d453a9257fcff87a8c5af P 96513fb661454619a601f5d1a4ad60a2: Deleting WAL directory at /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786501898713699-14748-0/minicluster-data/ts-2/wal/wals/b59b6f544f4d453a9257fcff87a8c5af
I20260812 02:36:12.640676 2258 tablet_replica.cc:330] T b59b6f544f4d453a9257fcff87a8c5af P 68989c4070784506b8b90d7e93027b94: stopping tablet replica
I20260812 02:36:12.640938 2260 ts_tablet_manager.cc:1939] T b59b6f544f4d453a9257fcff87a8c5af P 96513fb661454619a601f5d1a4ad60a2: Deleting consensus metadata
I20260812 02:36:12.641108 2258 raft_consensus.cc:2229] T b59b6f544f4d453a9257fcff87a8c5af P 68989c4070784506b8b90d7e93027b94 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 02:36:12.641535 2258 raft_consensus.cc:2258] T b59b6f544f4d453a9257fcff87a8c5af P 68989c4070784506b8b90d7e93027b94 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:36:12.642575 1668 catalog_manager.cc:4841] TS 96513fb661454619a601f5d1a4ad60a2 (127.14.103.3:39189): tablet b59b6f544f4d453a9257fcff87a8c5af (table test-table [id=27e16a04e8654d74b8cbd5b98a12eef8]) successfully deleted
I20260812 02:36:12.643289 2258 ts_tablet_manager.cc:1905] T b59b6f544f4d453a9257fcff87a8c5af P 68989c4070784506b8b90d7e93027b94: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260812 02:36:12.644470 1670 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.103.1:40853 (error 108)
I20260812 02:36:12.645526 2224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:12.645196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.1:40853 (local address 127.0.0.1:36608)
0812 02:36:12.645336 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0812 02:36:12.645419 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.1:40853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260812 02:36:12.645663 14748 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbep4M4/build/debug/bin/kudu with pid 1909
W20260812 02:36:12.645835 1670 catalog_manager.cc:4584] TS ca6a34962ea2410696f5a0a12631d074 (127.14.103.1:40853): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b59b6f544f4d453a9257fcff87a8c5af: Network error: Client connection negotiation failed: client connection to 127.14.103.1:40853: connect: Connection refused (error 111)
I20260812 02:36:12.646075 2258 ts_tablet_manager.cc:1918] T b59b6f544f4d453a9257fcff87a8c5af P 68989c4070784506b8b90d7e93027b94: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 02:36:12.653115 14748 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbep4M4/build/debug/bin/kudu with pid 2065
I20260812 02:36:12.657876 2224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:12.653408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.2:44039 (local address 127.0.0.1:38056)
0812 02:36:12.657646 (+ 4238us) negotiation.cc:107] Waiting for socket to connect
0812 02:36:12.657747 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.2:44039: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":4106}
I20260812 02:36:12.662767 14748 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbep4M4/build/debug/bin/kudu with pid 1650
2026-08-12T02:36:12Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2903 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-08-12T02:36:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T02:36:12Z Disabled control of system clock
I20260812 02:36:12.699612 14748 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbep4M4/build/debug/bin/kudu
/tmp/dist-test-taskbep4M4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786501898713699-14748-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786501898713699-14748-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786501898713699-14748-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786501898713699-14748-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.103.62:40233
--webserver_interface=127.14.103.62
--webserver_port=0
--builtin_ntp_servers=127.14.103.20:44383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.103.62:40233
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260812 02:36:12.817442 2265 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:36:12.817744 2265 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:36:12.817793 2265 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:36:12.822053 2265 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:36:12.822140 2265 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:36:12.822156 2265 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:36:12.822173 2265 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:36:12.822189 2265 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 02:36:12.827466 2265 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.103.20:44383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786501898713699-14748-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786501898713699-14748-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.103.62:40233
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.103.62:40233
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786501898713699-14748-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.103.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2265
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786501898713699-14748-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:36:12.828708 2265 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:36:12.830209 2265 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:36:12.839815 2271 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:36:12.839782 2270 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:36:12.839910 2273 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:36:12.840081 2265 server_base.cc:1034] running on GCE node
I20260812 02:36:12.840819 2265 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:36:12.842092 2265 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:36:12.843276 2265 hybrid_clock.cc:648] HybridClock initialized: now 1786502172843252 us; error 35 us; skew 500 ppm
I20260812 02:36:12.845875 2265 webserver.cc:460] Webserver started at http://127.14.103.62:34419/ using document root <none> and password file <none>
I20260812 02:36:12.846549 2265 fs_manager.cc:359] Metadata directory not provided
I20260812 02:36:12.846657 2265 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:36:12.846920 2265 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:36:12.850327 2265 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786501898713699-14748-0/minicluster-data/master-0/data/instance:
uuid: "21bb9a05a6e9470d8c7b27924df7dc7a"
format_stamp: "Formatted at 2026-08-12 02:36:12 on dist-test-slave-2w3w"
server_key: "9437beca70a2eb531240f7668ad79251"
server_key_iv: "c57ad2422b4832eb86d785625a7f2b50"
server_key_version: "encryptionkey@0"
I20260812 02:36:12.851217 2265 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786501898713699-14748-0/minicluster-data/master-0/wal/instance:
uuid: "21bb9a05a6e9470d8c7b27924df7dc7a"
format_stamp: "Formatted at 2026-08-12 02:36:12 on dist-test-slave-2w3w"
server_key: "9437beca70a2eb531240f7668ad79251"
server_key_iv: "c57ad2422b4832eb86d785625a7f2b50"
server_key_version: "encryptionkey@0"
I20260812 02:36:12.856146 2265 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260812 02:36:12.859191 2279 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:36:12.860653 2265 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260812 02:36:12.860837 2265 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786501898713699-14748-0/minicluster-data/master-0/data,/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786501898713699-14748-0/minicluster-data/master-0/wal
uuid: "21bb9a05a6e9470d8c7b27924df7dc7a"
format_stamp: "Formatted at 2026-08-12 02:36:12 on dist-test-slave-2w3w"
server_key: "9437beca70a2eb531240f7668ad79251"
server_key_iv: "c57ad2422b4832eb86d785625a7f2b50"
server_key_version: "encryptionkey@0"
I20260812 02:36:12.860983 2265 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786501898713699-14748-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786501898713699-14748-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786501898713699-14748-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:36:12.874029 2265 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:36:12.874811 2265 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:36:12.875087 2265 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:36:12.882170 2265 rpc_server.cc:307] RPC server started. Bound to: 127.14.103.62:40233
I20260812 02:36:12.882179 2342 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.103.62:40233 every 8 connection(s)
I20260812 02:36:12.883303 2265 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786501898713699-14748-0/minicluster-data/master-0/data/info.pb
I20260812 02:36:12.886617 2343 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:36:12.886713 14748 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbep4M4/build/debug/bin/kudu as pid 2265
I20260812 02:36:12.886924 14748 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786501898713699-14748-0/minicluster-data/master-0/wal/instance
I20260812 02:36:12.887234 14748 external_mini_cluster.cc:1428] Setting key be1d94e05a88c179386add4ca0fdb87b
I20260812 02:36:12.893764 2343 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 21bb9a05a6e9470d8c7b27924df7dc7a: Bootstrap starting.
I20260812 02:36:12.896425 2343 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 21bb9a05a6e9470d8c7b27924df7dc7a: Neither blocks nor log segments found. Creating new log.
I20260812 02:36:12.897020 2346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:12.888581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36652 (local address 127.14.103.62:40233)
0812 02:36:12.889247 (+ 666us) server_negotiation.cc:206] Beginning negotiation
0812 02:36:12.889262 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0812 02:36:12.889300 (+ 38us) server_negotiation.cc:407] Connection header received
0812 02:36:12.890018 (+ 718us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:36:12.890034 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:36:12.890429 (+ 395us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:36:12.890704 (+ 275us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:36:12.891110 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:12.892085 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:12.893305 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:12.893790 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:12.893954 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:36:12.895010 (+ 1056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:36:12.895016 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:36:12.895031 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:36:12.895608 (+ 577us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:36:12.895669 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:36:12.895690 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:36:12.895771 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0812 02:36:12.895941 (+ 170us) server_negotiation.cc:299] Negotiation successful
0812 02:36:12.896246 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":170,"threads_started":1}
I20260812 02:36:12.897274 2343 log.cc:824] T 00000000000000000000000000000000 P 21bb9a05a6e9470d8c7b27924df7dc7a: Log is configured to *not* fsync() on all Append() calls
I20260812 02:36:12.899384 2343 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 21bb9a05a6e9470d8c7b27924df7dc7a: No bootstrap required, opened a new log
I20260812 02:36:12.903053 2343 raft_consensus.cc:348] T 00000000000000000000000000000000 P 21bb9a05a6e9470d8c7b27924df7dc7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21bb9a05a6e9470d8c7b27924df7dc7a" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 40233 } }
I20260812 02:36:12.903280 2343 raft_consensus.cc:374] T 00000000000000000000000000000000 P 21bb9a05a6e9470d8c7b27924df7dc7a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:36:12.903322 2343 raft_consensus.cc:729] T 00000000000000000000000000000000 P 21bb9a05a6e9470d8c7b27924df7dc7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21bb9a05a6e9470d8c7b27924df7dc7a, State: Initialized, Role: FOLLOWER
I20260812 02:36:12.903678 2343 consensus_queue.cc:260] T 00000000000000000000000000000000 P 21bb9a05a6e9470d8c7b27924df7dc7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21bb9a05a6e9470d8c7b27924df7dc7a" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 40233 } }
I20260812 02:36:12.903793 2343 raft_consensus.cc:388] T 00000000000000000000000000000000 P 21bb9a05a6e9470d8c7b27924df7dc7a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 02:36:12.903846 2343 raft_consensus.cc:482] T 00000000000000000000000000000000 P 21bb9a05a6e9470d8c7b27924df7dc7a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 02:36:12.903915 2343 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 21bb9a05a6e9470d8c7b27924df7dc7a [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:36:12.904842 2343 raft_consensus.cc:504] T 00000000000000000000000000000000 P 21bb9a05a6e9470d8c7b27924df7dc7a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21bb9a05a6e9470d8c7b27924df7dc7a" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 40233 } }
I20260812 02:36:12.905143 2343 leader_election.cc:302] T 00000000000000000000000000000000 P 21bb9a05a6e9470d8c7b27924df7dc7a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 21bb9a05a6e9470d8c7b27924df7dc7a; no voters:
I20260812 02:36:12.905406 2343 leader_election.cc:288] T 00000000000000000000000000000000 P 21bb9a05a6e9470d8c7b27924df7dc7a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 02:36:12.905579 2348 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 21bb9a05a6e9470d8c7b27924df7dc7a [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:36:12.905889 2348 raft_consensus.cc:686] T 00000000000000000000000000000000 P 21bb9a05a6e9470d8c7b27924df7dc7a [term 1 LEADER]: Becoming Leader. State: Replica: 21bb9a05a6e9470d8c7b27924df7dc7a, State: Running, Role: LEADER
I20260812 02:36:12.906234 2348 consensus_queue.cc:237] T 00000000000000000000000000000000 P 21bb9a05a6e9470d8c7b27924df7dc7a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21bb9a05a6e9470d8c7b27924df7dc7a" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 40233 } }
I20260812 02:36:12.906695 2343 sys_catalog.cc:564] T 00000000000000000000000000000000 P 21bb9a05a6e9470d8c7b27924df7dc7a [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:36:12.907950 2349 sys_catalog.cc:455] T 00000000000000000000000000000000 P 21bb9a05a6e9470d8c7b27924df7dc7a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "21bb9a05a6e9470d8c7b27924df7dc7a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21bb9a05a6e9470d8c7b27924df7dc7a" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 40233 } } }
I20260812 02:36:12.908080 2349 sys_catalog.cc:458] T 00000000000000000000000000000000 P 21bb9a05a6e9470d8c7b27924df7dc7a [sys.catalog]: This master's current role is: LEADER
I20260812 02:36:12.908417 2351 sys_catalog.cc:455] T 00000000000000000000000000000000 P 21bb9a05a6e9470d8c7b27924df7dc7a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 21bb9a05a6e9470d8c7b27924df7dc7a. Latest consensus state: current_term: 1 leader_uuid: "21bb9a05a6e9470d8c7b27924df7dc7a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21bb9a05a6e9470d8c7b27924df7dc7a" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 40233 } } }
I20260812 02:36:12.908494 2359 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 02:36:12.908528 2351 sys_catalog.cc:458] T 00000000000000000000000000000000 P 21bb9a05a6e9470d8c7b27924df7dc7a [sys.catalog]: This master's current role is: LEADER
I20260812 02:36:12.911741 2359 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 02:36:12.917412 2359 catalog_manager.cc:1347] Generated new cluster ID: 2fe8aed7d270463eb764925638a7d4da
I20260812 02:36:12.917500 2359 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 02:36:12.934680 2359 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 02:36:12.936053 2359 catalog_manager.cc:1504] Loading token signing keys...
I20260812 02:36:12.946859 2359 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 21bb9a05a6e9470d8c7b27924df7dc7a: Generated new TSK 0
I20260812 02:36:12.947796 2359 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 02:36:12.963587 14748 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbep4M4/build/debug/bin/kudu
/tmp/dist-test-taskbep4M4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786501898713699-14748-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786501898713699-14748-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786501898713699-14748-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786501898713699-14748-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.103.1:0
--local_ip_for_outbound_sockets=127.14.103.1
--webserver_interface=127.14.103.1
--webserver_port=0
--tserver_master_addrs=127.14.103.62:40233
--builtin_ntp_servers=127.14.103.20:44383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260812 02:36:13.087388 2370 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:36:13.087687 2370 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:36:13.087729 2370 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:36:13.092089 2370 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:36:13.092162 2370 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:36:13.092223 2370 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.103.1
I20260812 02:36:13.097272 2370 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.103.20:44383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786501898713699-14748-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786501898713699-14748-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.103.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786501898713699-14748-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.103.1
--webserver_port=0
--tserver_master_addrs=127.14.103.62:40233
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2370
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.103.1
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786501898713699-14748-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:36:13.098522 2370 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:36:13.100055 2370 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260812 02:36:13.110574 2370 server_base.cc:1034] running on GCE node
W20260812 02:36:13.110512 2375 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:36:13.110508 2376 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:36:13.110538 2378 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:36:13.111353 2370 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:36:13.112018 2370 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:36:13.113234 2370 hybrid_clock.cc:648] HybridClock initialized: now 1786502173113222 us; error 26 us; skew 500 ppm
I20260812 02:36:13.115466 2370 webserver.cc:460] Webserver started at http://127.14.103.1:40799/ using document root <none> and password file <none>
I20260812 02:36:13.116101 2370 fs_manager.cc:359] Metadata directory not provided
I20260812 02:36:13.116200 2370 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:36:13.116453 2370 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:36:13.118616 2370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786501898713699-14748-0/minicluster-data/ts-0/data/instance:
uuid: "a5a8a2e3821345a2aad624f3ae714458"
format_stamp: "Formatted at 2026-08-12 02:36:13 on dist-test-slave-2w3w"
server_key: "c83e474c474e8c1765b890bce9cf4c59"
server_key_iv: "ec108a4668bc1db984acb18b73da354e"
server_key_version: "encryptionkey@0"
I20260812 02:36:13.119254 2370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786501898713699-14748-0/minicluster-data/ts-0/wal/instance:
uuid: "a5a8a2e3821345a2aad624f3ae714458"
format_stamp: "Formatted at 2026-08-12 02:36:13 on dist-test-slave-2w3w"
server_key: "c83e474c474e8c1765b890bce9cf4c59"
server_key_iv: "ec108a4668bc1db984acb18b73da354e"
server_key_version: "encryptionkey@0"
I20260812 02:36:13.123238 2370 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260812 02:36:13.125842 2386 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:36:13.127403 2370 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 02:36:13.127549 2370 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786501898713699-14748-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786501898713699-14748-0/minicluster-data/ts-0/wal
uuid: "a5a8a2e3821345a2aad624f3ae714458"
format_stamp: "Formatted at 2026-08-12 02:36:13 on dist-test-slave-2w3w"
server_key: "c83e474c474e8c1765b890bce9cf4c59"
server_key_iv: "ec108a4668bc1db984acb18b73da354e"
server_key_version: "encryptionkey@0"
I20260812 02:36:13.127677 2370 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786501898713699-14748-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786501898713699-14748-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786501898713699-14748-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:36:13.140763 2370 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:36:13.141538 2370 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:36:13.141772 2370 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:36:13.142423 2370 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:36:13.143455 2370 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:36:13.143529 2370 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:36:13.143601 2370 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:36:13.143651 2370 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:36:13.153152 2370 rpc_server.cc:307] RPC server started. Bound to: 127.14.103.1:34165
I20260812 02:36:13.153182 2513 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.103.1:34165 every 8 connection(s)
I20260812 02:36:13.154094 2370 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786501898713699-14748-0/minicluster-data/ts-0/data/info.pb
I20260812 02:36:13.162293 14748 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbep4M4/build/debug/bin/kudu as pid 2370
I20260812 02:36:13.162415 14748 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786501898713699-14748-0/minicluster-data/ts-0/wal/instance
I20260812 02:36:13.162758 14748 external_mini_cluster.cc:1428] Setting key e2146d666d64a63d4f92ba96c3e56673
I20260812 02:36:13.163997 2346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:13.156045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.1:41771 (local address 127.14.103.62:40233)
0812 02:36:13.156189 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0812 02:36:13.156194 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:36:13.157173 (+ 979us) server_negotiation.cc:407] Connection header received
0812 02:36:13.158637 (+ 1464us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:36:13.158641 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:36:13.158703 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:36:13.158806 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:36:13.159870 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:13.160430 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:13.161621 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:13.161872 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:13.161990 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:36:13.163407 (+ 1417us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:36:13.163414 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:36:13.163420 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:36:13.163439 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:36:13.163465 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:36:13.163476 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:36:13.163553 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0812 02:36:13.163768 (+ 215us) server_negotiation.cc:299] Negotiation successful
0812 02:36:13.163843 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260812 02:36:13.165323 2516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:13.156312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:40233 (local address 127.14.103.1:41771)
0812 02:36:13.156999 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0812 02:36:13.157042 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0812 02:36:13.158398 (+ 1356us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:36:13.159009 (+ 611us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:36:13.159017 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:36:13.159103 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:36:13.159712 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:36:13.159727 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:36:13.160570 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:36:13.160579 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:36:13.161483 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:36:13.161492 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:36:13.162037 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:36:13.162041 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:36:13.162217 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:36:13.163057 (+ 840us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 02:36:13.163059 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:36:13.163119 (+ 60us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 02:36:13.163212 (+ 93us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:36:13.163578 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:36:13.163590 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:36:13.163596 (+ 6us) client_negotiation.cc:769] Sending connection context
0812 02:36:13.163759 (+ 163us) client_negotiation.cc:240] Negotiation successful
0812 02:36:13.164150 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":124,"threads_started":1}
I20260812 02:36:13.165486 14748 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbep4M4/build/debug/bin/kudu
/tmp/dist-test-taskbep4M4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786501898713699-14748-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786501898713699-14748-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786501898713699-14748-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786501898713699-14748-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.103.2:0
--local_ip_for_outbound_sockets=127.14.103.2
--webserver_interface=127.14.103.2
--webserver_port=0
--tserver_master_addrs=127.14.103.62:40233
--builtin_ntp_servers=127.14.103.20:44383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260812 02:36:13.166932 2514 heartbeater.cc:347] Connected to a master server at 127.14.103.62:40233
I20260812 02:36:13.167256 2514 heartbeater.cc:464] Registering TS with master...
I20260812 02:36:13.167834 2514 heartbeater.cc:511] Master 127.14.103.62:40233 requested a full tablet report, sending...
I20260812 02:36:13.169638 2297 ts_manager.cc:194] Registered new tserver with Master: a5a8a2e3821345a2aad624f3ae714458 (127.14.103.1:34165)
I20260812 02:36:13.171377 2297 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.103.1:41771
W20260812 02:36:13.282582 2517 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:36:13.282887 2517 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:36:13.282930 2517 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:36:13.287190 2517 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:36:13.287272 2517 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:36:13.287333 2517 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.103.2
I20260812 02:36:13.292279 2517 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.103.20:44383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786501898713699-14748-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786501898713699-14748-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.103.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786501898713699-14748-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.103.2
--webserver_port=0
--tserver_master_addrs=127.14.103.62:40233
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2517
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.103.2
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786501898713699-14748-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:36:13.293371 2517 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:36:13.294809 2517 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:36:13.304800 2523 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:36:13.304811 2522 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:36:13.304921 2517 server_base.cc:1034] running on GCE node
W20260812 02:36:13.304833 2525 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:36:13.305483 2517 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:36:13.306151 2517 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:36:13.307337 2517 hybrid_clock.cc:648] HybridClock initialized: now 1786502173307315 us; error 41 us; skew 500 ppm
I20260812 02:36:13.309438 2517 webserver.cc:460] Webserver started at http://127.14.103.2:37805/ using document root <none> and password file <none>
I20260812 02:36:13.310031 2517 fs_manager.cc:359] Metadata directory not provided
I20260812 02:36:13.310127 2517 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:36:13.310357 2517 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:36:13.312546 2517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786501898713699-14748-0/minicluster-data/ts-1/data/instance:
uuid: "407b87698e054365922117f829351b9f"
format_stamp: "Formatted at 2026-08-12 02:36:13 on dist-test-slave-2w3w"
server_key: "151c02d7f64d663861b382e599d4938b"
server_key_iv: "78e951d9b3d09406aa70d1e4152725f3"
server_key_version: "encryptionkey@0"
I20260812 02:36:13.313160 2517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786501898713699-14748-0/minicluster-data/ts-1/wal/instance:
uuid: "407b87698e054365922117f829351b9f"
format_stamp: "Formatted at 2026-08-12 02:36:13 on dist-test-slave-2w3w"
server_key: "151c02d7f64d663861b382e599d4938b"
server_key_iv: "78e951d9b3d09406aa70d1e4152725f3"
server_key_version: "encryptionkey@0"
I20260812 02:36:13.317212 2517 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260812 02:36:13.319821 2531 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:36:13.321125 2517 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 02:36:13.321282 2517 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786501898713699-14748-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786501898713699-14748-0/minicluster-data/ts-1/wal
uuid: "407b87698e054365922117f829351b9f"
format_stamp: "Formatted at 2026-08-12 02:36:13 on dist-test-slave-2w3w"
server_key: "151c02d7f64d663861b382e599d4938b"
server_key_iv: "78e951d9b3d09406aa70d1e4152725f3"
server_key_version: "encryptionkey@0"
I20260812 02:36:13.321429 2517 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786501898713699-14748-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786501898713699-14748-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786501898713699-14748-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:36:13.339459 2517 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:36:13.340181 2517 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:36:13.340399 2517 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:36:13.341009 2517 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:36:13.341975 2517 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:36:13.342047 2517 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:36:13.342121 2517 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:36:13.342164 2517 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:36:13.351482 2517 rpc_server.cc:307] RPC server started. Bound to: 127.14.103.2:35467
I20260812 02:36:13.351522 2664 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.103.2:35467 every 8 connection(s)
I20260812 02:36:13.352452 2517 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786501898713699-14748-0/minicluster-data/ts-1/data/info.pb
I20260812 02:36:13.354794 14748 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbep4M4/build/debug/bin/kudu as pid 2517
I20260812 02:36:13.354946 14748 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786501898713699-14748-0/minicluster-data/ts-1/wal/instance
I20260812 02:36:13.355326 14748 external_mini_cluster.cc:1428] Setting key 3f3628fddc674c124b99a8cfb3feb9a1
I20260812 02:36:13.357558 14748 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbep4M4/build/debug/bin/kudu
/tmp/dist-test-taskbep4M4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786501898713699-14748-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786501898713699-14748-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786501898713699-14748-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786501898713699-14748-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.103.3:0
--local_ip_for_outbound_sockets=127.14.103.3
--webserver_interface=127.14.103.3
--webserver_port=0
--tserver_master_addrs=127.14.103.62:40233
--builtin_ntp_servers=127.14.103.20:44383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260812 02:36:13.363704 2346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:13.354333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.2:59573 (local address 127.14.103.62:40233)
0812 02:36:13.354477 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0812 02:36:13.354481 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:36:13.356001 (+ 1520us) server_negotiation.cc:407] Connection header received
0812 02:36:13.357529 (+ 1528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:36:13.357534 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:36:13.357596 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:36:13.357695 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:36:13.358689 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:13.359578 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:13.361338 (+ 1760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:13.361751 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:13.361873 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:36:13.363052 (+ 1179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:36:13.363058 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:36:13.363063 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:36:13.363084 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:36:13.363115 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:36:13.363130 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:36:13.363223 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0812 02:36:13.363384 (+ 161us) server_negotiation.cc:299] Negotiation successful
0812 02:36:13.363470 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260812 02:36:13.364543 2667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:13.354843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:40233 (local address 127.14.103.2:59573)
0812 02:36:13.355800 (+ 957us) negotiation.cc:107] Waiting for socket to connect
0812 02:36:13.355856 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0812 02:36:13.357300 (+ 1444us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:36:13.357942 (+ 642us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:36:13.357949 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:36:13.358036 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:36:13.358543 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:36:13.358556 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:36:13.359761 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:36:13.359767 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:36:13.361195 (+ 1428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:36:13.361206 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:36:13.361915 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:36:13.361918 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:36:13.362060 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:36:13.362726 (+ 666us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 02:36:13.362727 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:36:13.362769 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 02:36:13.362840 (+ 71us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:36:13.363225 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:36:13.363232 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:36:13.363237 (+ 5us) client_negotiation.cc:769] Sending connection context
0812 02:36:13.363353 (+ 116us) client_negotiation.cc:240] Negotiation successful
0812 02:36:13.363590 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":158,"threads_started":1}
I20260812 02:36:13.365907 2665 heartbeater.cc:347] Connected to a master server at 127.14.103.62:40233
I20260812 02:36:13.366287 2665 heartbeater.cc:464] Registering TS with master...
I20260812 02:36:13.367079 2665 heartbeater.cc:511] Master 127.14.103.62:40233 requested a full tablet report, sending...
I20260812 02:36:13.368350 2297 ts_manager.cc:194] Registered new tserver with Master: 407b87698e054365922117f829351b9f (127.14.103.2:35467)
I20260812 02:36:13.368979 2297 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.103.2:59573
W20260812 02:36:13.476225 2668 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:36:13.476537 2668 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:36:13.476578 2668 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:36:13.480777 2668 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:36:13.480844 2668 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:36:13.480909 2668 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.103.3
I20260812 02:36:13.485780 2668 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.103.20:44383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786501898713699-14748-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786501898713699-14748-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.103.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786501898713699-14748-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.103.3
--webserver_port=0
--tserver_master_addrs=127.14.103.62:40233
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2668
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.103.3
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786501898713699-14748-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:36:13.486927 2668 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:36:13.488353 2668 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:36:13.498751 2679 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:36:13.498736 2674 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:36:13.498736 2676 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:36:13.499044 2668 server_base.cc:1034] running on GCE node
I20260812 02:36:13.499517 2668 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:36:13.500195 2668 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:36:13.501386 2668 hybrid_clock.cc:648] HybridClock initialized: now 1786502173501353 us; error 48 us; skew 500 ppm
I20260812 02:36:13.503482 2668 webserver.cc:460] Webserver started at http://127.14.103.3:45477/ using document root <none> and password file <none>
I20260812 02:36:13.504091 2668 fs_manager.cc:359] Metadata directory not provided
I20260812 02:36:13.504155 2668 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:36:13.504381 2668 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:36:13.506484 2668 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786501898713699-14748-0/minicluster-data/ts-2/data/instance:
uuid: "a7dd71a87d7b4670bd8dca3bc7df7d43"
format_stamp: "Formatted at 2026-08-12 02:36:13 on dist-test-slave-2w3w"
server_key: "6c3b6db89a74fbff59aacf38f3f2a577"
server_key_iv: "acd1be5ddd3016d5a9f3d7c35d746ad8"
server_key_version: "encryptionkey@0"
I20260812 02:36:13.507128 2668 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786501898713699-14748-0/minicluster-data/ts-2/wal/instance:
uuid: "a7dd71a87d7b4670bd8dca3bc7df7d43"
format_stamp: "Formatted at 2026-08-12 02:36:13 on dist-test-slave-2w3w"
server_key: "6c3b6db89a74fbff59aacf38f3f2a577"
server_key_iv: "acd1be5ddd3016d5a9f3d7c35d746ad8"
server_key_version: "encryptionkey@0"
I20260812 02:36:13.511734 2668 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260812 02:36:13.514441 2686 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:36:13.515987 2668 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 02:36:13.516139 2668 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786501898713699-14748-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786501898713699-14748-0/minicluster-data/ts-2/wal
uuid: "a7dd71a87d7b4670bd8dca3bc7df7d43"
format_stamp: "Formatted at 2026-08-12 02:36:13 on dist-test-slave-2w3w"
server_key: "6c3b6db89a74fbff59aacf38f3f2a577"
server_key_iv: "acd1be5ddd3016d5a9f3d7c35d746ad8"
server_key_version: "encryptionkey@0"
I20260812 02:36:13.516268 2668 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786501898713699-14748-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786501898713699-14748-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786501898713699-14748-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:36:13.545143 2668 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:36:13.545922 2668 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:36:13.546137 2668 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:36:13.546763 2668 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:36:13.547832 2668 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:36:13.547905 2668 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:36:13.547974 2668 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:36:13.548027 2668 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:36:13.556878 2668 rpc_server.cc:307] RPC server started. Bound to: 127.14.103.3:37023
I20260812 02:36:13.556906 2823 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.103.3:37023 every 8 connection(s)
I20260812 02:36:13.557775 2668 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786501898713699-14748-0/minicluster-data/ts-2/data/info.pb
I20260812 02:36:13.566515 14748 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbep4M4/build/debug/bin/kudu as pid 2668
I20260812 02:36:13.566679 14748 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786501898713699-14748-0/minicluster-data/ts-2/wal/instance
I20260812 02:36:13.567099 14748 external_mini_cluster.cc:1428] Setting key 46114792b05ed1d57380e512d9d88f5d
I20260812 02:36:13.567408 2346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:13.559710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.3:54231 (local address 127.14.103.62:40233)
0812 02:36:13.559856 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0812 02:36:13.559860 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:36:13.560964 (+ 1104us) server_negotiation.cc:407] Connection header received
0812 02:36:13.562345 (+ 1381us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:36:13.562348 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:36:13.562401 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:36:13.562482 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:36:13.563470 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:13.564036 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:13.565285 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:13.565522 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:13.565604 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:36:13.566843 (+ 1239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:36:13.566847 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:36:13.566850 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:36:13.566866 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:36:13.566892 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:36:13.566903 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:36:13.567020 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0812 02:36:13.567198 (+ 178us) server_negotiation.cc:299] Negotiation successful
0812 02:36:13.567264 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260812 02:36:13.568476 2826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:13.560086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:40233 (local address 127.14.103.3:54231)
0812 02:36:13.560790 (+ 704us) negotiation.cc:107] Waiting for socket to connect
0812 02:36:13.560832 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0812 02:36:13.562130 (+ 1298us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:36:13.562645 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:36:13.562651 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:36:13.562735 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:36:13.563335 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:36:13.563347 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:36:13.564143 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:36:13.564145 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:36:13.565183 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:36:13.565192 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:36:13.565695 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:36:13.565698 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:36:13.565856 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:36:13.566534 (+ 678us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 02:36:13.566536 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:36:13.566576 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 02:36:13.566663 (+ 87us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:36:13.567041 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:36:13.567048 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:36:13.567053 (+ 5us) client_negotiation.cc:769] Sending connection context
0812 02:36:13.567174 (+ 121us) client_negotiation.cc:240] Negotiation successful
0812 02:36:13.567473 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":176,"threads_started":1}
I20260812 02:36:13.569581 2824 heartbeater.cc:347] Connected to a master server at 127.14.103.62:40233
I20260812 02:36:13.569823 2824 heartbeater.cc:464] Registering TS with master...
I20260812 02:36:13.570374 2824 heartbeater.cc:511] Master 127.14.103.62:40233 requested a full tablet report, sending...
I20260812 02:36:13.571480 2296 ts_manager.cc:194] Registered new tserver with Master: a7dd71a87d7b4670bd8dca3bc7df7d43 (127.14.103.3:37023)
I20260812 02:36:13.572110 2296 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.103.3:54231
I20260812 02:36:13.581359 14748 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 02:36:13.587420 2346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:13.583464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36654 (local address 127.14.103.62:40233)
0812 02:36:13.583630 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0812 02:36:13.583635 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:36:13.583828 (+ 193us) server_negotiation.cc:407] Connection header received
0812 02:36:13.583958 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:36:13.583963 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:36:13.584015 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:36:13.584103 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:36:13.584396 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:13.584914 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:13.585776 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:13.586002 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:13.586114 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:36:13.586950 (+ 836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:36:13.586983 (+ 33us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:36:13.586988 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:36:13.587002 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:36:13.587023 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:36:13.587039 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:36:13.587119 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0812 02:36:13.587212 (+ 93us) server_negotiation.cc:299] Negotiation successful
0812 02:36:13.587266 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260812 02:36:13.590745 2296 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36654:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 02:36:13.593900 2296 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 02:36:13.609606 2837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:13.602932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.3:37023 (local address 127.0.0.1:59720)
0812 02:36:13.603808 (+ 876us) negotiation.cc:107] Waiting for socket to connect
0812 02:36:13.603847 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0812 02:36:13.603953 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:36:13.604792 (+ 839us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:36:13.604801 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:36:13.604825 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:36:13.605006 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:36:13.605014 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:36:13.606168 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:36:13.606175 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:36:13.607735 (+ 1560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:36:13.607767 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:36:13.608940 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:36:13.608943 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:36:13.609096 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:36:13.609151 (+ 55us) client_negotiation.cc:769] Sending connection context
0812 02:36:13.609259 (+ 108us) client_negotiation.cc:240] Negotiation successful
0812 02:36:13.609414 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":706,"mutex_wait_us":47,"thread_start_us":87,"threads_started":1}
I20260812 02:36:13.609611 2839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:13.603228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58964 (local address 127.14.103.1:34165)
0812 02:36:13.604490 (+ 1262us) server_negotiation.cc:206] Beginning negotiation
0812 02:36:13.604502 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 02:36:13.604524 (+ 22us) server_negotiation.cc:407] Connection header received
0812 02:36:13.604593 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:36:13.604597 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:36:13.604789 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:36:13.604940 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:36:13.605222 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:13.606474 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:13.608065 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:13.608784 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:13.608915 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:36:13.609140 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0812 02:36:13.609349 (+ 209us) server_negotiation.cc:299] Negotiation successful
0812 02:36:13.609465 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":987,"thread_start_us":156,"threads_started":1}
I20260812 02:36:13.609766 2842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:13.603839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47300 (local address 127.14.103.2:35467)
0812 02:36:13.604201 (+ 362us) server_negotiation.cc:206] Beginning negotiation
0812 02:36:13.604215 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0812 02:36:13.604578 (+ 363us) server_negotiation.cc:407] Connection header received
0812 02:36:13.604664 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:36:13.604669 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:36:13.604888 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:36:13.605052 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:36:13.605330 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:13.606466 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:13.608074 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:13.608810 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:13.608922 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:36:13.609138 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0812 02:36:13.609495 (+ 357us) server_negotiation.cc:299] Negotiation successful
0812 02:36:13.609594 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":254,"thread_start_us":137,"threads_started":1}
I20260812 02:36:13.610005 2838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:13.603143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59720 (local address 127.14.103.3:37023)
0812 02:36:13.604282 (+ 1139us) server_negotiation.cc:206] Beginning negotiation
0812 02:36:13.604293 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 02:36:13.604317 (+ 24us) server_negotiation.cc:407] Connection header received
0812 02:36:13.604394 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:36:13.604400 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:36:13.604612 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:36:13.605163 (+ 551us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:36:13.605319 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:13.606044 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:13.607896 (+ 1852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:13.608789 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:13.609518 (+ 729us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:36:13.609697 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0812 02:36:13.609743 (+ 46us) server_negotiation.cc:299] Negotiation successful
0812 02:36:13.609842 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1025,"thread_start_us":184,"threads_started":1}
I20260812 02:36:13.610323 2840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:13.602950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.1:34165 (local address 127.0.0.1:58964)
0812 02:36:13.603873 (+ 923us) negotiation.cc:107] Waiting for socket to connect
0812 02:36:13.603889 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 02:36:13.603994 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:36:13.604929 (+ 935us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:36:13.604932 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:36:13.604956 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:36:13.605078 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:36:13.605086 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:36:13.606669 (+ 1583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:36:13.606673 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:36:13.607900 (+ 1227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:36:13.607910 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:36:13.608918 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:36:13.608922 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:36:13.609047 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:36:13.609161 (+ 114us) client_negotiation.cc:769] Sending connection context
0812 02:36:13.609312 (+ 151us) client_negotiation.cc:240] Negotiation successful
0812 02:36:13.609406 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":858,"mutex_wait_us":88,"thread_start_us":98,"threads_started":1}
I20260812 02:36:13.610349 2841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:13.603326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.2:35467 (local address 127.0.0.1:47300)
0812 02:36:13.604350 (+ 1024us) negotiation.cc:107] Waiting for socket to connect
0812 02:36:13.604383 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0812 02:36:13.604483 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:36:13.605117 (+ 634us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:36:13.605120 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:36:13.605138 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:36:13.605211 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:36:13.605216 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:36:13.606584 (+ 1368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:36:13.606587 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:36:13.607913 (+ 1326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:36:13.607927 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:36:13.609200 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:36:13.609203 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:36:13.609290 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:36:13.609308 (+ 18us) client_negotiation.cc:769] Sending connection context
0812 02:36:13.609371 (+ 63us) client_negotiation.cc:240] Negotiation successful
0812 02:36:13.609425 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":972,"mutex_wait_us":99,"thread_start_us":100,"threads_started":1}
I20260812 02:36:13.613029 2437 tablet_service.cc:1467] Processing CreateTablet for tablet a4c6bb11b78645d09fcf4848f892904d (DEFAULT_TABLE table=test-table [id=2a7a5419063e4e68987df707f6a2f017]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:36:13.613085 2587 tablet_service.cc:1467] Processing CreateTablet for tablet a4c6bb11b78645d09fcf4848f892904d (DEFAULT_TABLE table=test-table [id=2a7a5419063e4e68987df707f6a2f017]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:36:13.613083 2742 tablet_service.cc:1467] Processing CreateTablet for tablet a4c6bb11b78645d09fcf4848f892904d (DEFAULT_TABLE table=test-table [id=2a7a5419063e4e68987df707f6a2f017]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:36:13.614096 2437 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4c6bb11b78645d09fcf4848f892904d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:36:13.614133 2742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4c6bb11b78645d09fcf4848f892904d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:36:13.614133 2587 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4c6bb11b78645d09fcf4848f892904d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:36:13.621114 2843 tablet_bootstrap.cc:492] T a4c6bb11b78645d09fcf4848f892904d P a5a8a2e3821345a2aad624f3ae714458: Bootstrap starting.
I20260812 02:36:13.622501 2844 tablet_bootstrap.cc:492] T a4c6bb11b78645d09fcf4848f892904d P 407b87698e054365922117f829351b9f: Bootstrap starting.
I20260812 02:36:13.622536 2845 tablet_bootstrap.cc:492] T a4c6bb11b78645d09fcf4848f892904d P a7dd71a87d7b4670bd8dca3bc7df7d43: Bootstrap starting.
I20260812 02:36:13.623667 2843 tablet_bootstrap.cc:654] T a4c6bb11b78645d09fcf4848f892904d P a5a8a2e3821345a2aad624f3ae714458: Neither blocks nor log segments found. Creating new log.
I20260812 02:36:13.624423 2844 tablet_bootstrap.cc:654] T a4c6bb11b78645d09fcf4848f892904d P 407b87698e054365922117f829351b9f: Neither blocks nor log segments found. Creating new log.
I20260812 02:36:13.624579 2843 log.cc:824] T a4c6bb11b78645d09fcf4848f892904d P a5a8a2e3821345a2aad624f3ae714458: Log is configured to *not* fsync() on all Append() calls
I20260812 02:36:13.624629 2845 tablet_bootstrap.cc:654] T a4c6bb11b78645d09fcf4848f892904d P a7dd71a87d7b4670bd8dca3bc7df7d43: Neither blocks nor log segments found. Creating new log.
I20260812 02:36:13.625447 2844 log.cc:824] T a4c6bb11b78645d09fcf4848f892904d P 407b87698e054365922117f829351b9f: Log is configured to *not* fsync() on all Append() calls
I20260812 02:36:13.625488 2845 log.cc:824] T a4c6bb11b78645d09fcf4848f892904d P a7dd71a87d7b4670bd8dca3bc7df7d43: Log is configured to *not* fsync() on all Append() calls
I20260812 02:36:13.627089 2843 tablet_bootstrap.cc:492] T a4c6bb11b78645d09fcf4848f892904d P a5a8a2e3821345a2aad624f3ae714458: No bootstrap required, opened a new log
I20260812 02:36:13.627310 2843 ts_tablet_manager.cc:1397] T a4c6bb11b78645d09fcf4848f892904d P a5a8a2e3821345a2aad624f3ae714458: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.002s
I20260812 02:36:13.627470 2845 tablet_bootstrap.cc:492] T a4c6bb11b78645d09fcf4848f892904d P a7dd71a87d7b4670bd8dca3bc7df7d43: No bootstrap required, opened a new log
I20260812 02:36:13.627585 2844 tablet_bootstrap.cc:492] T a4c6bb11b78645d09fcf4848f892904d P 407b87698e054365922117f829351b9f: No bootstrap required, opened a new log
I20260812 02:36:13.627666 2845 ts_tablet_manager.cc:1397] T a4c6bb11b78645d09fcf4848f892904d P a7dd71a87d7b4670bd8dca3bc7df7d43: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260812 02:36:13.627784 2844 ts_tablet_manager.cc:1397] T a4c6bb11b78645d09fcf4848f892904d P 407b87698e054365922117f829351b9f: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260812 02:36:13.631079 2844 raft_consensus.cc:348] T a4c6bb11b78645d09fcf4848f892904d P 407b87698e054365922117f829351b9f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7dd71a87d7b4670bd8dca3bc7df7d43" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 37023 } } peers { permanent_uuid: "a5a8a2e3821345a2aad624f3ae714458" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 34165 } } peers { permanent_uuid: "407b87698e054365922117f829351b9f" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 35467 } }
I20260812 02:36:13.631405 2844 raft_consensus.cc:374] T a4c6bb11b78645d09fcf4848f892904d P 407b87698e054365922117f829351b9f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:36:13.631465 2844 raft_consensus.cc:729] T a4c6bb11b78645d09fcf4848f892904d P 407b87698e054365922117f829351b9f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 407b87698e054365922117f829351b9f, State: Initialized, Role: FOLLOWER
I20260812 02:36:13.631990 2844 consensus_queue.cc:260] T a4c6bb11b78645d09fcf4848f892904d P 407b87698e054365922117f829351b9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7dd71a87d7b4670bd8dca3bc7df7d43" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 37023 } } peers { permanent_uuid: "a5a8a2e3821345a2aad624f3ae714458" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 34165 } } peers { permanent_uuid: "407b87698e054365922117f829351b9f" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 35467 } }
I20260812 02:36:13.631916 2843 raft_consensus.cc:348] T a4c6bb11b78645d09fcf4848f892904d P a5a8a2e3821345a2aad624f3ae714458 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7dd71a87d7b4670bd8dca3bc7df7d43" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 37023 } } peers { permanent_uuid: "a5a8a2e3821345a2aad624f3ae714458" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 34165 } } peers { permanent_uuid: "407b87698e054365922117f829351b9f" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 35467 } }
I20260812 02:36:13.631963 2845 raft_consensus.cc:348] T a4c6bb11b78645d09fcf4848f892904d P a7dd71a87d7b4670bd8dca3bc7df7d43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7dd71a87d7b4670bd8dca3bc7df7d43" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 37023 } } peers { permanent_uuid: "a5a8a2e3821345a2aad624f3ae714458" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 34165 } } peers { permanent_uuid: "407b87698e054365922117f829351b9f" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 35467 } }
I20260812 02:36:13.632184 2843 raft_consensus.cc:374] T a4c6bb11b78645d09fcf4848f892904d P a5a8a2e3821345a2aad624f3ae714458 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:36:13.632184 2845 raft_consensus.cc:374] T a4c6bb11b78645d09fcf4848f892904d P a7dd71a87d7b4670bd8dca3bc7df7d43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:36:13.632234 2843 raft_consensus.cc:729] T a4c6bb11b78645d09fcf4848f892904d P a5a8a2e3821345a2aad624f3ae714458 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5a8a2e3821345a2aad624f3ae714458, State: Initialized, Role: FOLLOWER
I20260812 02:36:13.632244 2845 raft_consensus.cc:729] T a4c6bb11b78645d09fcf4848f892904d P a7dd71a87d7b4670bd8dca3bc7df7d43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7dd71a87d7b4670bd8dca3bc7df7d43, State: Initialized, Role: FOLLOWER
I20260812 02:36:13.632642 2843 consensus_queue.cc:260] T a4c6bb11b78645d09fcf4848f892904d P a5a8a2e3821345a2aad624f3ae714458 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7dd71a87d7b4670bd8dca3bc7df7d43" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 37023 } } peers { permanent_uuid: "a5a8a2e3821345a2aad624f3ae714458" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 34165 } } peers { permanent_uuid: "407b87698e054365922117f829351b9f" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 35467 } }
I20260812 02:36:13.632663 2845 consensus_queue.cc:260] T a4c6bb11b78645d09fcf4848f892904d P a7dd71a87d7b4670bd8dca3bc7df7d43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7dd71a87d7b4670bd8dca3bc7df7d43" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 37023 } } peers { permanent_uuid: "a5a8a2e3821345a2aad624f3ae714458" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 34165 } } peers { permanent_uuid: "407b87698e054365922117f829351b9f" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 35467 } }
I20260812 02:36:13.633345 2844 ts_tablet_manager.cc:1428] T a4c6bb11b78645d09fcf4848f892904d P 407b87698e054365922117f829351b9f: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260812 02:36:13.633553 2843 ts_tablet_manager.cc:1428] T a4c6bb11b78645d09fcf4848f892904d P a5a8a2e3821345a2aad624f3ae714458: Time spent starting tablet: real 0.006s user 0.004s sys 0.002s
I20260812 02:36:13.633553 2845 ts_tablet_manager.cc:1428] T a4c6bb11b78645d09fcf4848f892904d P a7dd71a87d7b4670bd8dca3bc7df7d43: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260812 02:36:13.633728 2665 heartbeater.cc:503] Master 127.14.103.62:40233 was elected leader, sending a full tablet report...
I20260812 02:36:13.633893 2514 heartbeater.cc:503] Master 127.14.103.62:40233 was elected leader, sending a full tablet report...
I20260812 02:36:13.634121 2824 heartbeater.cc:503] Master 127.14.103.62:40233 was elected leader, sending a full tablet report...
W20260812 02:36:13.655403 2515 tablet.cc:2369] T a4c6bb11b78645d09fcf4848f892904d P a5a8a2e3821345a2aad624f3ae714458: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 02:36:13.808758 2825 tablet.cc:2369] T a4c6bb11b78645d09fcf4848f892904d P a7dd71a87d7b4670bd8dca3bc7df7d43: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 02:36:13.821239 2851 raft_consensus.cc:482] T a4c6bb11b78645d09fcf4848f892904d P a5a8a2e3821345a2aad624f3ae714458 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 02:36:13.821501 2851 raft_consensus.cc:504] T a4c6bb11b78645d09fcf4848f892904d P a5a8a2e3821345a2aad624f3ae714458 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7dd71a87d7b4670bd8dca3bc7df7d43" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 37023 } } peers { permanent_uuid: "a5a8a2e3821345a2aad624f3ae714458" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 34165 } } peers { permanent_uuid: "407b87698e054365922117f829351b9f" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 35467 } }
I20260812 02:36:13.822767 2851 leader_election.cc:288] T a4c6bb11b78645d09fcf4848f892904d P a5a8a2e3821345a2aad624f3ae714458 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a7dd71a87d7b4670bd8dca3bc7df7d43 (127.14.103.3:37023), 407b87698e054365922117f829351b9f (127.14.103.2:35467)
I20260812 02:36:13.827375 2838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:13.823368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.1:59817 (local address 127.14.103.3:37023)
0812 02:36:13.823604 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0812 02:36:13.823610 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 02:36:13.823629 (+ 19us) server_negotiation.cc:407] Connection header received
0812 02:36:13.823827 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:36:13.823830 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:36:13.823881 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:36:13.823972 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:36:13.824253 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:13.824796 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:13.826208 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:13.826645 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:13.826782 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:36:13.826845 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0812 02:36:13.827186 (+ 341us) server_negotiation.cc:299] Negotiation successful
0812 02:36:13.827264 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152}
I20260812 02:36:13.827359 2842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:13.823332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.1:50507 (local address 127.14.103.2:35467)
0812 02:36:13.823508 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0812 02:36:13.823513 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:36:13.823799 (+ 286us) server_negotiation.cc:407] Connection header received
0812 02:36:13.823967 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:36:13.823971 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:36:13.824015 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:36:13.824096 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:36:13.824391 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:13.824885 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:13.826197 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:13.826653 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:13.826788 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:36:13.826844 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0812 02:36:13.827129 (+ 285us) server_negotiation.cc:299] Negotiation successful
0812 02:36:13.827231 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260812 02:36:13.827380 2855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:13.823220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.2:35467 (local address 127.14.103.1:50507)
0812 02:36:13.823692 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0812 02:36:13.823705 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 02:36:13.823842 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:36:13.824119 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:36:13.824122 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:36:13.824137 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:36:13.824281 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:36:13.824286 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:36:13.825049 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:36:13.825052 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:36:13.826066 (+ 1014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:36:13.826077 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:36:13.826783 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:36:13.826787 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:36:13.826919 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:36:13.827004 (+ 85us) client_negotiation.cc:769] Sending connection context
0812 02:36:13.827093 (+ 89us) client_negotiation.cc:240] Negotiation successful
0812 02:36:13.827191 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":415,"mutex_wait_us":81,"thread_start_us":121,"threads_started":1}
I20260812 02:36:13.827785 2854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:13.823220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.3:37023 (local address 127.14.103.1:59817)
0812 02:36:13.823558 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0812 02:36:13.823574 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 02:36:13.823696 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:36:13.823995 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:36:13.823998 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:36:13.824014 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:36:13.824143 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:36:13.824149 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:36:13.824912 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:36:13.824916 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:36:13.826070 (+ 1154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:36:13.826102 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:36:13.826781 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:36:13.826785 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:36:13.827015 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:36:13.827038 (+ 23us) client_negotiation.cc:769] Sending connection context
0812 02:36:13.827591 (+ 553us) client_negotiation.cc:240] Negotiation successful
0812 02:36:13.827677 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"mutex_wait_us":149,"thread_start_us":110,"threads_started":1}
I20260812 02:36:13.828449 2610 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4c6bb11b78645d09fcf4848f892904d" candidate_uuid: "a5a8a2e3821345a2aad624f3ae714458" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "407b87698e054365922117f829351b9f" is_pre_election: true
I20260812 02:36:13.828467 2765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4c6bb11b78645d09fcf4848f892904d" candidate_uuid: "a5a8a2e3821345a2aad624f3ae714458" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7dd71a87d7b4670bd8dca3bc7df7d43" is_pre_election: true
I20260812 02:36:13.828775 2765 raft_consensus.cc:2454] T a4c6bb11b78645d09fcf4848f892904d P a7dd71a87d7b4670bd8dca3bc7df7d43 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5a8a2e3821345a2aad624f3ae714458 in term 0.
I20260812 02:36:13.828780 2610 raft_consensus.cc:2454] T a4c6bb11b78645d09fcf4848f892904d P 407b87698e054365922117f829351b9f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5a8a2e3821345a2aad624f3ae714458 in term 0.
I20260812 02:36:13.829389 2389 leader_election.cc:302] T a4c6bb11b78645d09fcf4848f892904d P a5a8a2e3821345a2aad624f3ae714458 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 407b87698e054365922117f829351b9f, a5a8a2e3821345a2aad624f3ae714458, a7dd71a87d7b4670bd8dca3bc7df7d43; no voters:
I20260812 02:36:13.829710 2851 raft_consensus.cc:2781] T a4c6bb11b78645d09fcf4848f892904d P a5a8a2e3821345a2aad624f3ae714458 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 02:36:13.829802 2851 raft_consensus.cc:482] T a4c6bb11b78645d09fcf4848f892904d P a5a8a2e3821345a2aad624f3ae714458 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 02:36:13.829847 2851 raft_consensus.cc:3037] T a4c6bb11b78645d09fcf4848f892904d P a5a8a2e3821345a2aad624f3ae714458 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:36:13.830953 2851 raft_consensus.cc:504] T a4c6bb11b78645d09fcf4848f892904d P a5a8a2e3821345a2aad624f3ae714458 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7dd71a87d7b4670bd8dca3bc7df7d43" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 37023 } } peers { permanent_uuid: "a5a8a2e3821345a2aad624f3ae714458" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 34165 } } peers { permanent_uuid: "407b87698e054365922117f829351b9f" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 35467 } }
I20260812 02:36:13.831519 2851 leader_election.cc:288] T a4c6bb11b78645d09fcf4848f892904d P a5a8a2e3821345a2aad624f3ae714458 [CANDIDATE]: Term 1 election: Requested vote from peers a7dd71a87d7b4670bd8dca3bc7df7d43 (127.14.103.3:37023), 407b87698e054365922117f829351b9f (127.14.103.2:35467)
I20260812 02:36:13.831943 2610 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4c6bb11b78645d09fcf4848f892904d" candidate_uuid: "a5a8a2e3821345a2aad624f3ae714458" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "407b87698e054365922117f829351b9f"
I20260812 02:36:13.831943 2765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4c6bb11b78645d09fcf4848f892904d" candidate_uuid: "a5a8a2e3821345a2aad624f3ae714458" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7dd71a87d7b4670bd8dca3bc7df7d43"
I20260812 02:36:13.832103 2765 raft_consensus.cc:3037] T a4c6bb11b78645d09fcf4848f892904d P a7dd71a87d7b4670bd8dca3bc7df7d43 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:36:13.832103 2610 raft_consensus.cc:3037] T a4c6bb11b78645d09fcf4848f892904d P 407b87698e054365922117f829351b9f [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:36:13.833323 2765 raft_consensus.cc:2454] T a4c6bb11b78645d09fcf4848f892904d P a7dd71a87d7b4670bd8dca3bc7df7d43 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5a8a2e3821345a2aad624f3ae714458 in term 1.
I20260812 02:36:13.833379 2610 raft_consensus.cc:2454] T a4c6bb11b78645d09fcf4848f892904d P 407b87698e054365922117f829351b9f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5a8a2e3821345a2aad624f3ae714458 in term 1.
I20260812 02:36:13.833706 2389 leader_election.cc:302] T a4c6bb11b78645d09fcf4848f892904d P a5a8a2e3821345a2aad624f3ae714458 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a5a8a2e3821345a2aad624f3ae714458, a7dd71a87d7b4670bd8dca3bc7df7d43; no voters:
I20260812 02:36:13.833936 2851 raft_consensus.cc:2781] T a4c6bb11b78645d09fcf4848f892904d P a5a8a2e3821345a2aad624f3ae714458 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:36:13.834151 2851 raft_consensus.cc:686] T a4c6bb11b78645d09fcf4848f892904d P a5a8a2e3821345a2aad624f3ae714458 [term 1 LEADER]: Becoming Leader. State: Replica: a5a8a2e3821345a2aad624f3ae714458, State: Running, Role: LEADER
I20260812 02:36:13.834525 2851 consensus_queue.cc:237] T a4c6bb11b78645d09fcf4848f892904d P a5a8a2e3821345a2aad624f3ae714458 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7dd71a87d7b4670bd8dca3bc7df7d43" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 37023 } } peers { permanent_uuid: "a5a8a2e3821345a2aad624f3ae714458" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 34165 } } peers { permanent_uuid: "407b87698e054365922117f829351b9f" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 35467 } }
I20260812 02:36:13.838060 2296 catalog_manager.cc:5495] T a4c6bb11b78645d09fcf4848f892904d P a5a8a2e3821345a2aad624f3ae714458 reported cstate change: term changed from 0 to 1, leader changed from <none> to a5a8a2e3821345a2aad624f3ae714458 (127.14.103.1). New cstate: current_term: 1 leader_uuid: "a5a8a2e3821345a2aad624f3ae714458" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7dd71a87d7b4670bd8dca3bc7df7d43" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 37023 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5a8a2e3821345a2aad624f3ae714458" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 34165 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "407b87698e054365922117f829351b9f" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 35467 } health_report { overall_health: UNKNOWN } } }
W20260812 02:36:13.853960 2666 tablet.cc:2369] T a4c6bb11b78645d09fcf4848f892904d P 407b87698e054365922117f829351b9f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 02:36:13.904104 2839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:13.900244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58968 (local address 127.14.103.1:34165)
0812 02:36:13.900392 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0812 02:36:13.900397 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:36:13.900432 (+ 35us) server_negotiation.cc:407] Connection header received
0812 02:36:13.900512 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:36:13.900514 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:36:13.900572 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:36:13.900656 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 02:36:13.900973 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:13.901479 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:13.902398 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:13.902620 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:13.902707 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:36:13.903173 (+ 466us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 02:36:13.903625 (+ 452us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 02:36:13.903722 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0812 02:36:13.903870 (+ 148us) server_negotiation.cc:299] Negotiation successful
0812 02:36:13.903971 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260812 02:36:13.910467 2765 raft_consensus.cc:1261] T a4c6bb11b78645d09fcf4848f892904d P a7dd71a87d7b4670bd8dca3bc7df7d43 [term 1 FOLLOWER]: Refusing update from remote peer a5a8a2e3821345a2aad624f3ae714458: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:36:13.910467 2610 raft_consensus.cc:1261] T a4c6bb11b78645d09fcf4848f892904d P 407b87698e054365922117f829351b9f [term 1 FOLLOWER]: Refusing update from remote peer a5a8a2e3821345a2aad624f3ae714458: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:36:13.911267 2856 consensus_queue.cc:1035] T a4c6bb11b78645d09fcf4848f892904d P a5a8a2e3821345a2aad624f3ae714458 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7dd71a87d7b4670bd8dca3bc7df7d43" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 37023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:36:13.911477 2851 consensus_queue.cc:1035] T a4c6bb11b78645d09fcf4848f892904d P a5a8a2e3821345a2aad624f3ae714458 [LEADER]: Connected to new peer: Peer: permanent_uuid: "407b87698e054365922117f829351b9f" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 35467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:36:13.918882 2859 mvcc.cc:205] Tried to move back new op lower bound from 7317512904329437184 to 7317512904029667328. Current Snapshot: MvccSnapshot[applied={T|T < 7317512904329437184}]
I20260812 02:36:13.919353 2860 mvcc.cc:205] Tried to move back new op lower bound from 7317512904329437184 to 7317512904029667328. Current Snapshot: MvccSnapshot[applied={T|T < 7317512904329437184}]
I20260812 02:36:13.929510 2296 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36654:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 02:36:13.929723 2296 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36654:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 02:36:13.932622 2296 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 21bb9a05a6e9470d8c7b27924df7dc7a: Sending DeleteTablet for 3 replicas of tablet a4c6bb11b78645d09fcf4848f892904d
I20260812 02:36:13.933632 2437 tablet_service.cc:1514] Processing DeleteTablet for tablet a4c6bb11b78645d09fcf4848f892904d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:36:13 UTC) from {username='slave'} at 127.0.0.1:58964
I20260812 02:36:13.933655 2742 tablet_service.cc:1514] Processing DeleteTablet for tablet a4c6bb11b78645d09fcf4848f892904d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:36:13 UTC) from {username='slave'} at 127.0.0.1:59720
I20260812 02:36:13.933808 2587 tablet_service.cc:1514] Processing DeleteTablet for tablet a4c6bb11b78645d09fcf4848f892904d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:36:13 UTC) from {username='slave'} at 127.0.0.1:47300
I20260812 02:36:13.934244 2870 tablet_replica.cc:330] T a4c6bb11b78645d09fcf4848f892904d P a7dd71a87d7b4670bd8dca3bc7df7d43: stopping tablet replica
I20260812 02:36:13.934598 2870 raft_consensus.cc:2229] T a4c6bb11b78645d09fcf4848f892904d P a7dd71a87d7b4670bd8dca3bc7df7d43 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 02:36:13.934628 2869 tablet_replica.cc:330] T a4c6bb11b78645d09fcf4848f892904d P a5a8a2e3821345a2aad624f3ae714458: stopping tablet replica
I20260812 02:36:13.935001 2870 raft_consensus.cc:2258] T a4c6bb11b78645d09fcf4848f892904d P a7dd71a87d7b4670bd8dca3bc7df7d43 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:36:13.934988 2869 raft_consensus.cc:2229] T a4c6bb11b78645d09fcf4848f892904d P a5a8a2e3821345a2aad624f3ae714458 [term 1 LEADER]: Raft consensus shutting down.
I20260812 02:36:13.935433 14748 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbep4M4/build/debug/bin/kudu with pid 2370
I20260812 02:36:13.935415 2869 raft_consensus.cc:2258] T a4c6bb11b78645d09fcf4848f892904d P a5a8a2e3821345a2aad624f3ae714458 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:36:13.936909 2871 tablet_replica.cc:330] T a4c6bb11b78645d09fcf4848f892904d P 407b87698e054365922117f829351b9f: stopping tablet replica
I20260812 02:36:13.937247 2871 raft_consensus.cc:2229] T a4c6bb11b78645d09fcf4848f892904d P 407b87698e054365922117f829351b9f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 02:36:13.937570 2871 raft_consensus.cc:2258] T a4c6bb11b78645d09fcf4848f892904d P 407b87698e054365922117f829351b9f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:36:13.939888 2871 ts_tablet_manager.cc:1905] T a4c6bb11b78645d09fcf4848f892904d P 407b87698e054365922117f829351b9f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:36:13.940937 2870 ts_tablet_manager.cc:1905] T a4c6bb11b78645d09fcf4848f892904d P a7dd71a87d7b4670bd8dca3bc7df7d43: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:36:13.943034 2871 ts_tablet_manager.cc:1918] T a4c6bb11b78645d09fcf4848f892904d P 407b87698e054365922117f829351b9f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 02:36:13.943152 2871 log.cc:1196] T a4c6bb11b78645d09fcf4848f892904d P 407b87698e054365922117f829351b9f: Deleting WAL directory at /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786501898713699-14748-0/minicluster-data/ts-1/wal/wals/a4c6bb11b78645d09fcf4848f892904d
I20260812 02:36:13.943519 2871 ts_tablet_manager.cc:1939] T a4c6bb11b78645d09fcf4848f892904d P 407b87698e054365922117f829351b9f: Deleting consensus metadata
I20260812 02:36:13.943917 2870 ts_tablet_manager.cc:1918] T a4c6bb11b78645d09fcf4848f892904d P a7dd71a87d7b4670bd8dca3bc7df7d43: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 02:36:13.944024 2870 log.cc:1196] T a4c6bb11b78645d09fcf4848f892904d P a7dd71a87d7b4670bd8dca3bc7df7d43: Deleting WAL directory at /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786501898713699-14748-0/minicluster-data/ts-2/wal/wals/a4c6bb11b78645d09fcf4848f892904d
I20260812 02:36:13.944329 2870 ts_tablet_manager.cc:1939] T a4c6bb11b78645d09fcf4848f892904d P a7dd71a87d7b4670bd8dca3bc7df7d43: Deleting consensus metadata
I20260812 02:36:13.945161 2283 catalog_manager.cc:4841] TS 407b87698e054365922117f829351b9f (127.14.103.2:35467): tablet a4c6bb11b78645d09fcf4848f892904d (table test-table [id=2a7a5419063e4e68987df707f6a2f017]) successfully deleted
I20260812 02:36:13.945780 2282 catalog_manager.cc:4841] TS a7dd71a87d7b4670bd8dca3bc7df7d43 (127.14.103.3:37023): tablet a4c6bb11b78645d09fcf4848f892904d (table test-table [id=2a7a5419063e4e68987df707f6a2f017]) successfully deleted
W20260812 02:36:13.946113 2281 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.103.1:34165 (error 108)
I20260812 02:36:13.947027 14748 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbep4M4/build/debug/bin/kudu with pid 2517
I20260812 02:36:13.947474 2841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:13.947058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.1:34165 (local address 127.0.0.1:58978)
0812 02:36:13.947285 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0812 02:36:13.947371 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.1:34165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127}
W20260812 02:36:13.947639 2281 catalog_manager.cc:4584] TS a5a8a2e3821345a2aad624f3ae714458 (127.14.103.1:34165): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a4c6bb11b78645d09fcf4848f892904d: Network error: Client connection negotiation failed: client connection to 127.14.103.1:34165: connect: Connection refused (error 111)
I20260812 02:36:13.956530 14748 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbep4M4/build/debug/bin/kudu with pid 2668
I20260812 02:36:13.966182 14748 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbep4M4/build/debug/bin/kudu with pid 2265
2026-08-12T02:36:13Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1302 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501898713699-14748-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 02:36:14 dist-test-slave-2w3w krb5kdc[2876](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 02:36:14 dist-test-slave-2w3w krb5kdc[2876](info): set up 2 sockets
Aug 12 02:36:14 dist-test-slave-2w3w krb5kdc[2876](info): commencing operation
krb5kdc: starting...
W20260812 02:36:15.102484 14748 mini_kdc.cc:121] Time spent starting KDC: real 1.110s user 0.006s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501898713699-14748-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501898713699-14748-0/krb5kdc/test-user.keytab.
Aug 12 02:36:15 dist-test-slave-2w3w krb5kdc[2876](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502175, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T02:36:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T02:36:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.103.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.103.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.103.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.62.keytab.
Entry for principal kudu/127.14.103.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.62.keytab.
I20260812 02:36:15.250896 14748 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbep4M4/build/debug/bin/kudu
/tmp/dist-test-taskbep4M4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501898713699-14748-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501898713699-14748-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501898713699-14748-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501898713699-14748-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.103.62:36313
--webserver_interface=127.14.103.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.62.keytab
--principal=kudu/127.14.103.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.103.20:38981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.103.62:36313
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501898713699-14748-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501898713699-14748-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501898713699-14748-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:36:15.364384 2890 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:36:15.364711 2890 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:36:15.364795 2890 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:36:15.368991 2890 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:36:15.369102 2890 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:36:15.369144 2890 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:36:15.369201 2890 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:36:15.369247 2890 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 02:36:15.374843 2890 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.103.20:38981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501898713699-14748-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501898713699-14748-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.103.62:36313
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.62.keytab
--principal=kudu/127.14.103.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.103.62:36313
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501898713699-14748-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.103.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2890
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501898713699-14748-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:36:15.376050 2890 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:36:15.377116 2890 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:36:15.386123 2896 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:36:15.386315 2890 server_base.cc:1034] running on GCE node
W20260812 02:36:15.386176 2897 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:36:15.386204 2900 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:36:15.387146 2890 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:36:15.388352 2890 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:36:15.389547 2890 hybrid_clock.cc:648] HybridClock initialized: now 1786502175389539 us; error 31 us; skew 500 ppm
Aug 12 02:36:15 dist-test-slave-2w3w krb5kdc[2876](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502175, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:36:15.397074 2890 init.cc:377] Logged in from keytab as kudu/127.14.103.62@KRBTEST.COM (short username kudu)
I20260812 02:36:15.398190 2890 webserver.cc:460] Webserver started at http://127.14.103.62:45161/ using document root <none> and password file <none>
I20260812 02:36:15.398801 2890 fs_manager.cc:359] Metadata directory not provided
I20260812 02:36:15.398901 2890 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:36:15.399186 2890 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:36:15.401409 2890 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501898713699-14748-0/minicluster-data/master-0/data/instance:
uuid: "ea3e7b556e00409fb52175828a07669a"
format_stamp: "Formatted at 2026-08-12 02:36:15 on dist-test-slave-2w3w"
server_key: "d97931ab8e9974dbb4d0ccd9a7a655f0"
server_key_iv: "0f9c6ec399f3689d0352189ea3ee2f4f"
server_key_version: "encryptionkey@0"
I20260812 02:36:15.402014 2890 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501898713699-14748-0/minicluster-data/master-0/wal/instance:
uuid: "ea3e7b556e00409fb52175828a07669a"
format_stamp: "Formatted at 2026-08-12 02:36:15 on dist-test-slave-2w3w"
server_key: "d97931ab8e9974dbb4d0ccd9a7a655f0"
server_key_iv: "0f9c6ec399f3689d0352189ea3ee2f4f"
server_key_version: "encryptionkey@0"
I20260812 02:36:15.406195 2890 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260812 02:36:15.408968 2908 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:36:15.410251 2890 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260812 02:36:15.410406 2890 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501898713699-14748-0/minicluster-data/master-0/data,/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501898713699-14748-0/minicluster-data/master-0/wal
uuid: "ea3e7b556e00409fb52175828a07669a"
format_stamp: "Formatted at 2026-08-12 02:36:15 on dist-test-slave-2w3w"
server_key: "d97931ab8e9974dbb4d0ccd9a7a655f0"
server_key_iv: "0f9c6ec399f3689d0352189ea3ee2f4f"
server_key_version: "encryptionkey@0"
I20260812 02:36:15.410540 2890 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501898713699-14748-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501898713699-14748-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501898713699-14748-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:36:15.438438 2890 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:36:15.442157 2890 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:36:15.442395 2890 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:36:15.450356 2890 rpc_server.cc:307] RPC server started. Bound to: 127.14.103.62:36313
I20260812 02:36:15.450376 2969 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.103.62:36313 every 8 connection(s)
I20260812 02:36:15.451421 2890 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501898713699-14748-0/minicluster-data/master-0/data/info.pb
I20260812 02:36:15.454288 2970 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:36:15.459033 14748 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbep4M4/build/debug/bin/kudu as pid 2890
I20260812 02:36:15.459177 14748 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501898713699-14748-0/minicluster-data/master-0/wal/instance
I20260812 02:36:15.459533 14748 external_mini_cluster.cc:1428] Setting key f3531b81a4b35ef19efae6f38d8c7fda
I20260812 02:36:15.460390 2970 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea3e7b556e00409fb52175828a07669a: Bootstrap starting.
I20260812 02:36:15.462808 2970 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ea3e7b556e00409fb52175828a07669a: Neither blocks nor log segments found. Creating new log.
I20260812 02:36:15.463621 2970 log.cc:824] T 00000000000000000000000000000000 P ea3e7b556e00409fb52175828a07669a: Log is configured to *not* fsync() on all Append() calls
I20260812 02:36:15.465963 2970 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea3e7b556e00409fb52175828a07669a: No bootstrap required, opened a new log
Aug 12 02:36:15 dist-test-slave-2w3w krb5kdc[2876](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502175, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.103.62@KRBTEST.COM
I20260812 02:36:15.470362 2970 raft_consensus.cc:348] T 00000000000000000000000000000000 P ea3e7b556e00409fb52175828a07669a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea3e7b556e00409fb52175828a07669a" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 36313 } }
I20260812 02:36:15.470618 2970 raft_consensus.cc:374] T 00000000000000000000000000000000 P ea3e7b556e00409fb52175828a07669a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:36:15.470701 2970 raft_consensus.cc:729] T 00000000000000000000000000000000 P ea3e7b556e00409fb52175828a07669a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea3e7b556e00409fb52175828a07669a, State: Initialized, Role: FOLLOWER
I20260812 02:36:15.471212 2970 consensus_queue.cc:260] T 00000000000000000000000000000000 P ea3e7b556e00409fb52175828a07669a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea3e7b556e00409fb52175828a07669a" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 36313 } }
I20260812 02:36:15.471371 2970 raft_consensus.cc:388] T 00000000000000000000000000000000 P ea3e7b556e00409fb52175828a07669a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 02:36:15.471462 2970 raft_consensus.cc:482] T 00000000000000000000000000000000 P ea3e7b556e00409fb52175828a07669a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 02:36:15.471560 2970 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ea3e7b556e00409fb52175828a07669a [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:36:15.472586 2970 raft_consensus.cc:504] T 00000000000000000000000000000000 P ea3e7b556e00409fb52175828a07669a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea3e7b556e00409fb52175828a07669a" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 36313 } }
I20260812 02:36:15.472949 2970 leader_election.cc:302] T 00000000000000000000000000000000 P ea3e7b556e00409fb52175828a07669a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ea3e7b556e00409fb52175828a07669a; no voters:
I20260812 02:36:15.473322 2970 leader_election.cc:288] T 00000000000000000000000000000000 P ea3e7b556e00409fb52175828a07669a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 02:36:15.473402 2977 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ea3e7b556e00409fb52175828a07669a [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:36:15.473599 2977 raft_consensus.cc:686] T 00000000000000000000000000000000 P ea3e7b556e00409fb52175828a07669a [term 1 LEADER]: Becoming Leader. State: Replica: ea3e7b556e00409fb52175828a07669a, State: Running, Role: LEADER
I20260812 02:36:15.473950 2977 consensus_queue.cc:237] T 00000000000000000000000000000000 P ea3e7b556e00409fb52175828a07669a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea3e7b556e00409fb52175828a07669a" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 36313 } }
I20260812 02:36:15.474632 2970 sys_catalog.cc:564] T 00000000000000000000000000000000 P ea3e7b556e00409fb52175828a07669a [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:36:15.475220 2977 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea3e7b556e00409fb52175828a07669a [sys.catalog]: SysCatalogTable state changed. Reason: New leader ea3e7b556e00409fb52175828a07669a. Latest consensus state: current_term: 1 leader_uuid: "ea3e7b556e00409fb52175828a07669a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea3e7b556e00409fb52175828a07669a" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 36313 } } }
I20260812 02:36:15.475438 2977 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea3e7b556e00409fb52175828a07669a [sys.catalog]: This master's current role is: LEADER
I20260812 02:36:15.475625 2978 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea3e7b556e00409fb52175828a07669a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ea3e7b556e00409fb52175828a07669a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea3e7b556e00409fb52175828a07669a" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 36313 } } }
I20260812 02:36:15.475723 2975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:15.461077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47836 (local address 127.14.103.62:36313)
0812 02:36:15.461533 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0812 02:36:15.461549 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0812 02:36:15.461587 (+ 38us) server_negotiation.cc:407] Connection header received
0812 02:36:15.462306 (+ 719us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:36:15.462324 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:36:15.462656 (+ 332us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:36:15.463030 (+ 374us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:36:15.463926 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:15.465317 (+ 1391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:15.466399 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:15.467098 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:15.467251 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:36:15.470111 (+ 2860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:36:15.470116 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:36:15.470131 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:36:15.470158 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:36:15.472188 (+ 2030us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:36:15.472770 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:36:15.472776 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:36:15.472819 (+ 43us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:36:15.472942 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:36:15.473263 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:36:15.473266 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:36:15.473268 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:36:15.473823 (+ 555us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:36:15.474031 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0812 02:36:15.474320 (+ 289us) server_negotiation.cc:299] Negotiation successful
0812 02:36:15.474656 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":130,"threads_started":1}
I20260812 02:36:15.475735 2978 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea3e7b556e00409fb52175828a07669a [sys.catalog]: This master's current role is: LEADER
I20260812 02:36:15.475950 2984 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 02:36:15.479378 2984 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 02:36:15.485706 2984 catalog_manager.cc:1347] Generated new cluster ID: 985ea48afa38489ca3ec5bd36ba02141
I20260812 02:36:15.485791 2984 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 02:36:15.494165 2984 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 02:36:15.495211 2984 catalog_manager.cc:1504] Loading token signing keys...
I20260812 02:36:15.505069 2984 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ea3e7b556e00409fb52175828a07669a: Generated new TSK 0
I20260812 02:36:15.505945 2984 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.103.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.103.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.103.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.1.keytab.
Entry for principal kudu/127.14.103.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.1.keytab.
I20260812 02:36:15.543285 14748 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbep4M4/build/debug/bin/kudu
/tmp/dist-test-taskbep4M4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501898713699-14748-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501898713699-14748-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501898713699-14748-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501898713699-14748-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.103.1:0
--local_ip_for_outbound_sockets=127.14.103.1
--webserver_interface=127.14.103.1
--webserver_port=0
--tserver_master_addrs=127.14.103.62:36313
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.1.keytab
--principal=kudu/127.14.103.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.103.20:38981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501898713699-14748-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501898713699-14748-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501898713699-14748-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:36:15.658362 3002 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:36:15.658679 3002 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:36:15.658725 3002 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:36:15.662756 3002 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:36:15.662823 3002 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:36:15.662884 3002 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.103.1
I20260812 02:36:15.668406 3002 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.103.20:38981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501898713699-14748-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501898713699-14748-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.1.keytab
--principal=kudu/127.14.103.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.103.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501898713699-14748-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.103.1
--webserver_port=0
--tserver_master_addrs=127.14.103.62:36313
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3002
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.103.1
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501898713699-14748-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:36:15.669498 3002 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:36:15.670580 3002 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:36:15.680800 3008 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:36:15.680883 3010 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:36:15.681133 3002 server_base.cc:1034] running on GCE node
W20260812 02:36:15.680810 3007 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:36:15.681823 3002 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:36:15.682524 3002 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:36:15.683784 3002 hybrid_clock.cc:648] HybridClock initialized: now 1786502175683763 us; error 37 us; skew 500 ppm
Aug 12 02:36:15 dist-test-slave-2w3w krb5kdc[2876](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502175, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:36:15.686833 3002 init.cc:377] Logged in from keytab as kudu/127.14.103.1@KRBTEST.COM (short username kudu)
I20260812 02:36:15.687809 3002 webserver.cc:460] Webserver started at http://127.14.103.1:45657/ using document root <none> and password file <none>
I20260812 02:36:15.688388 3002 fs_manager.cc:359] Metadata directory not provided
I20260812 02:36:15.688505 3002 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:36:15.688733 3002 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:36:15.690780 3002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501898713699-14748-0/minicluster-data/ts-0/data/instance:
uuid: "947908d10e4d423eb3d4fc5fd99a8b2f"
format_stamp: "Formatted at 2026-08-12 02:36:15 on dist-test-slave-2w3w"
server_key: "6af5996c2eaada0ddfb45e9361ae0448"
server_key_iv: "70ad18aaeaa43a6d6817c62603f41ebe"
server_key_version: "encryptionkey@0"
I20260812 02:36:15.691426 3002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501898713699-14748-0/minicluster-data/ts-0/wal/instance:
uuid: "947908d10e4d423eb3d4fc5fd99a8b2f"
format_stamp: "Formatted at 2026-08-12 02:36:15 on dist-test-slave-2w3w"
server_key: "6af5996c2eaada0ddfb45e9361ae0448"
server_key_iv: "70ad18aaeaa43a6d6817c62603f41ebe"
server_key_version: "encryptionkey@0"
I20260812 02:36:15.695264 3002 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260812 02:36:15.697670 3017 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:36:15.698908 3002 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260812 02:36:15.699071 3002 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501898713699-14748-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501898713699-14748-0/minicluster-data/ts-0/wal
uuid: "947908d10e4d423eb3d4fc5fd99a8b2f"
format_stamp: "Formatted at 2026-08-12 02:36:15 on dist-test-slave-2w3w"
server_key: "6af5996c2eaada0ddfb45e9361ae0448"
server_key_iv: "70ad18aaeaa43a6d6817c62603f41ebe"
server_key_version: "encryptionkey@0"
I20260812 02:36:15.699187 3002 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501898713699-14748-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501898713699-14748-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501898713699-14748-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:36:15.732671 3002 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:36:15.736133 3002 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:36:15.736351 3002 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:36:15.736976 3002 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:36:15.737812 3002 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:36:15.737884 3002 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:36:15.737955 3002 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:36:15.738001 3002 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:36:15.747207 3002 rpc_server.cc:307] RPC server started. Bound to: 127.14.103.1:38823
I20260812 02:36:15.747225 3146 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.103.1:38823 every 8 connection(s)
I20260812 02:36:15.748108 3002 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501898713699-14748-0/minicluster-data/ts-0/data/info.pb
I20260812 02:36:15.751539 14748 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbep4M4/build/debug/bin/kudu as pid 3002
I20260812 02:36:15.751652 14748 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501898713699-14748-0/minicluster-data/ts-0/wal/instance
I20260812 02:36:15.751963 14748 external_mini_cluster.cc:1428] Setting key 40dfb3460480f027f59e74b94b842e62
Aug 12 02:36:15 dist-test-slave-2w3w krb5kdc[2876](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502175, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.1@KRBTEST.COM for kudu/127.14.103.62@KRBTEST.COM
I20260812 02:36:15.763144 2975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:15.749987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.1:33945 (local address 127.14.103.62:36313)
0812 02:36:15.750182 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0812 02:36:15.750186 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:36:15.750988 (+ 802us) server_negotiation.cc:407] Connection header received
0812 02:36:15.752607 (+ 1619us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:36:15.752611 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:36:15.752666 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:36:15.752767 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:36:15.754229 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:15.754771 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:15.755988 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:15.756225 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:15.756342 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:36:15.759135 (+ 2793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:36:15.759139 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:36:15.759142 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:36:15.759175 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:36:15.760891 (+ 1716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:36:15.761407 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:36:15.761411 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:36:15.761413 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:36:15.761510 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:36:15.761849 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:36:15.761854 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:36:15.761857 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:36:15.762120 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:36:15.762234 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0812 02:36:15.762719 (+ 485us) server_negotiation.cc:299] Negotiation successful
0812 02:36:15.762883 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
WARNING: no policy specified for kudu/127.14.103.2@KRBTEST.COM; defaulting to no policy
I20260812 02:36:15.763849 3150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:15.750303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:36313 (local address 127.14.103.1:33945)
0812 02:36:15.750798 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0812 02:36:15.750843 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0812 02:36:15.752368 (+ 1525us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:36:15.752969 (+ 601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:36:15.752977 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:36:15.753557 (+ 580us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:36:15.754071 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:36:15.754085 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:36:15.754909 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:36:15.754912 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:36:15.755859 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:36:15.755868 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:36:15.756384 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:36:15.756388 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:36:15.756562 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:36:15.757186 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:36:15.757211 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:36:15.758911 (+ 1700us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:36:15.761016 (+ 2105us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:36:15.761021 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:36:15.761032 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:36:15.761286 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:36:15.761602 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:36:15.761604 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:36:15.761605 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:36:15.761738 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:36:15.762227 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:36:15.762232 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:36:15.762458 (+ 226us) client_negotiation.cc:769] Sending connection context
0812 02:36:15.762602 (+ 144us) client_negotiation.cc:240] Negotiation successful
0812 02:36:15.762853 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":127,"threads_started":1}
I20260812 02:36:15.765271 3147 heartbeater.cc:347] Connected to a master server at 127.14.103.62:36313
I20260812 02:36:15.765563 3147 heartbeater.cc:464] Registering TS with master...
I20260812 02:36:15.766096 3147 heartbeater.cc:511] Master 127.14.103.62:36313 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.103.2@KRBTEST.COM" created.
I20260812 02:36:15.767836 2927 ts_manager.cc:194] Registered new tserver with Master: 947908d10e4d423eb3d4fc5fd99a8b2f (127.14.103.1:38823)
I20260812 02:36:15.769299 2927 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.103.1@KRBTEST.COM'} at 127.14.103.1:33945
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.103.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.2.keytab.
Entry for principal kudu/127.14.103.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.2.keytab.
I20260812 02:36:15.787510 14748 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbep4M4/build/debug/bin/kudu
/tmp/dist-test-taskbep4M4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501898713699-14748-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501898713699-14748-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501898713699-14748-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501898713699-14748-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.103.2:0
--local_ip_for_outbound_sockets=127.14.103.2
--webserver_interface=127.14.103.2
--webserver_port=0
--tserver_master_addrs=127.14.103.62:36313
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.2.keytab
--principal=kudu/127.14.103.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.103.20:38981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501898713699-14748-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501898713699-14748-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501898713699-14748-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:36:15.901376 3155 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:36:15.901705 3155 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:36:15.901791 3155 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:36:15.906265 3155 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:36:15.906378 3155 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:36:15.906481 3155 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.103.2
I20260812 02:36:15.912202 3155 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.103.20:38981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501898713699-14748-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501898713699-14748-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.2.keytab
--principal=kudu/127.14.103.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.103.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501898713699-14748-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.103.2
--webserver_port=0
--tserver_master_addrs=127.14.103.62:36313
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3155
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.103.2
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501898713699-14748-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:36:15.913318 3155 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:36:15.914342 3155 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:36:15.924180 3161 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:36:15.924183 3164 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:36:15.924422 3155 server_base.cc:1034] running on GCE node
W20260812 02:36:15.924183 3162 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:36:15.925020 3155 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:36:15.925702 3155 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:36:15.926908 3155 hybrid_clock.cc:648] HybridClock initialized: now 1786502175926865 us; error 52 us; skew 500 ppm
Aug 12 02:36:15 dist-test-slave-2w3w krb5kdc[2876](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502175, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:36:15.929764 3155 init.cc:377] Logged in from keytab as kudu/127.14.103.2@KRBTEST.COM (short username kudu)
I20260812 02:36:15.930701 3155 webserver.cc:460] Webserver started at http://127.14.103.2:42119/ using document root <none> and password file <none>
I20260812 02:36:15.931372 3155 fs_manager.cc:359] Metadata directory not provided
I20260812 02:36:15.931470 3155 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:36:15.931706 3155 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:36:15.933769 3155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501898713699-14748-0/minicluster-data/ts-1/data/instance:
uuid: "2207d0e8f81d4c338903fe9c291ba2b1"
format_stamp: "Formatted at 2026-08-12 02:36:15 on dist-test-slave-2w3w"
server_key: "7375c7c620bd7a7e018e01272d56e613"
server_key_iv: "8f7604c42d5158f223397f2119b56ae2"
server_key_version: "encryptionkey@0"
I20260812 02:36:15.934372 3155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501898713699-14748-0/minicluster-data/ts-1/wal/instance:
uuid: "2207d0e8f81d4c338903fe9c291ba2b1"
format_stamp: "Formatted at 2026-08-12 02:36:15 on dist-test-slave-2w3w"
server_key: "7375c7c620bd7a7e018e01272d56e613"
server_key_iv: "8f7604c42d5158f223397f2119b56ae2"
server_key_version: "encryptionkey@0"
I20260812 02:36:15.938269 3155 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260812 02:36:15.940783 3172 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:36:15.942047 3155 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260812 02:36:15.942201 3155 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501898713699-14748-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501898713699-14748-0/minicluster-data/ts-1/wal
uuid: "2207d0e8f81d4c338903fe9c291ba2b1"
format_stamp: "Formatted at 2026-08-12 02:36:15 on dist-test-slave-2w3w"
server_key: "7375c7c620bd7a7e018e01272d56e613"
server_key_iv: "8f7604c42d5158f223397f2119b56ae2"
server_key_version: "encryptionkey@0"
I20260812 02:36:15.942327 3155 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501898713699-14748-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501898713699-14748-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501898713699-14748-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:36:15.979673 3155 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:36:15.982920 3155 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:36:15.983179 3155 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:36:15.983794 3155 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:36:15.984663 3155 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:36:15.984733 3155 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:36:15.984815 3155 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:36:15.984861 3155 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:36:15.993791 3155 rpc_server.cc:307] RPC server started. Bound to: 127.14.103.2:40011
I20260812 02:36:15.993809 3298 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.103.2:40011 every 8 connection(s)
I20260812 02:36:15.994693 3155 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501898713699-14748-0/minicluster-data/ts-1/data/info.pb
I20260812 02:36:15.995461 14748 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbep4M4/build/debug/bin/kudu as pid 3155
I20260812 02:36:15.995592 14748 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501898713699-14748-0/minicluster-data/ts-1/wal/instance
I20260812 02:36:15.995944 14748 external_mini_cluster.cc:1428] Setting key 595fedec0a9750542ba42b0d077ccc39
Aug 12 02:36:15 dist-test-slave-2w3w krb5kdc[2876](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502175, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.2@KRBTEST.COM for kudu/127.14.103.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.14.103.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.103.3@KRBTEST.COM" created.
I20260812 02:36:16.010136 2975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:15.996652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.2:39883 (local address 127.14.103.62:36313)
0812 02:36:15.996792 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0812 02:36:15.996796 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:36:15.997759 (+ 963us) server_negotiation.cc:407] Connection header received
0812 02:36:15.999203 (+ 1444us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:36:15.999207 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:36:15.999261 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:36:15.999367 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:36:16.000518 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:16.001059 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:16.002207 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:16.002501 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:16.002612 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:36:16.005712 (+ 3100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:36:16.005715 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:36:16.005718 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:36:16.005747 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:36:16.007651 (+ 1904us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:36:16.008353 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:36:16.008356 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:36:16.008357 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:36:16.008441 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:36:16.008803 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:36:16.008806 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:36:16.008807 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:36:16.009148 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:36:16.009233 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0812 02:36:16.009815 (+ 582us) server_negotiation.cc:299] Negotiation successful
0812 02:36:16.009927 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260812 02:36:16.011193 3302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:15.997104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:36313 (local address 127.14.103.2:39883)
0812 02:36:15.997620 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0812 02:36:15.997663 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0812 02:36:15.998989 (+ 1326us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:36:15.999541 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:36:15.999547 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:36:15.999905 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:36:16.000370 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:36:16.000382 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:36:16.001188 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:36:16.001191 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:36:16.002071 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:36:16.002079 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:36:16.002663 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:36:16.002668 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:36:16.002834 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:36:16.003540 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:36:16.003556 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:36:16.005477 (+ 1921us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:36:16.007888 (+ 2411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:36:16.007896 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:36:16.007917 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:36:16.008232 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:36:16.008554 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:36:16.008556 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:36:16.008557 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:36:16.008715 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:36:16.009274 (+ 559us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:36:16.009281 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:36:16.009570 (+ 289us) client_negotiation.cc:769] Sending connection context
0812 02:36:16.009791 (+ 221us) client_negotiation.cc:240] Negotiation successful
0812 02:36:16.010210 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":132,"threads_started":1}
I20260812 02:36:16.012452 3300 heartbeater.cc:347] Connected to a master server at 127.14.103.62:36313
I20260812 02:36:16.012727 3300 heartbeater.cc:464] Registering TS with master...
I20260812 02:36:16.013350 3300 heartbeater.cc:511] Master 127.14.103.62:36313 requested a full tablet report, sending...
I20260812 02:36:16.015228 2927 ts_manager.cc:194] Registered new tserver with Master: 2207d0e8f81d4c338903fe9c291ba2b1 (127.14.103.2:40011)
I20260812 02:36:16.016155 2927 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.103.2@KRBTEST.COM'} at 127.14.103.2:39883
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.103.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.3.keytab.
Entry for principal kudu/127.14.103.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.3.keytab.
I20260812 02:36:16.030705 14748 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbep4M4/build/debug/bin/kudu
/tmp/dist-test-taskbep4M4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501898713699-14748-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501898713699-14748-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501898713699-14748-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501898713699-14748-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.103.3:0
--local_ip_for_outbound_sockets=127.14.103.3
--webserver_interface=127.14.103.3
--webserver_port=0
--tserver_master_addrs=127.14.103.62:36313
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.3.keytab
--principal=kudu/127.14.103.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.103.20:38981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501898713699-14748-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501898713699-14748-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501898713699-14748-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:36:16.147128 3305 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:36:16.147452 3305 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:36:16.147531 3305 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:36:16.151676 3305 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:36:16.151782 3305 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:36:16.151885 3305 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.103.3
I20260812 02:36:16.157414 3305 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.103.20:38981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501898713699-14748-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501898713699-14748-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.3.keytab
--principal=kudu/127.14.103.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.103.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501898713699-14748-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.103.3
--webserver_port=0
--tserver_master_addrs=127.14.103.62:36313
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3305
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.103.3
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501898713699-14748-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:36:16.158528 3305 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:36:16.159870 3305 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:36:16.170202 3314 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:36:16.170320 3305 server_base.cc:1034] running on GCE node
W20260812 02:36:16.170238 3315 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:36:16.170197 3317 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:36:16.170948 3305 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:36:16.171675 3305 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:36:16.172853 3305 hybrid_clock.cc:648] HybridClock initialized: now 1786502176172823 us; error 47 us; skew 500 ppm
Aug 12 02:36:16 dist-test-slave-2w3w krb5kdc[2876](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502176, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:36:16.176059 3305 init.cc:377] Logged in from keytab as kudu/127.14.103.3@KRBTEST.COM (short username kudu)
I20260812 02:36:16.177044 3305 webserver.cc:460] Webserver started at http://127.14.103.3:38751/ using document root <none> and password file <none>
I20260812 02:36:16.177640 3305 fs_manager.cc:359] Metadata directory not provided
I20260812 02:36:16.177736 3305 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:36:16.177980 3305 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:36:16.180146 3305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501898713699-14748-0/minicluster-data/ts-2/data/instance:
uuid: "3552dd9467a749bbb1f161821fe54793"
format_stamp: "Formatted at 2026-08-12 02:36:16 on dist-test-slave-2w3w"
server_key: "8f95f878113d9935791bee9e1a61a870"
server_key_iv: "2843495802459045db5589ae8e712217"
server_key_version: "encryptionkey@0"
I20260812 02:36:16.180747 3305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501898713699-14748-0/minicluster-data/ts-2/wal/instance:
uuid: "3552dd9467a749bbb1f161821fe54793"
format_stamp: "Formatted at 2026-08-12 02:36:16 on dist-test-slave-2w3w"
server_key: "8f95f878113d9935791bee9e1a61a870"
server_key_iv: "2843495802459045db5589ae8e712217"
server_key_version: "encryptionkey@0"
I20260812 02:36:16.184747 3305 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260812 02:36:16.187234 3324 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:36:16.188485 3305 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260812 02:36:16.188635 3305 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501898713699-14748-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501898713699-14748-0/minicluster-data/ts-2/wal
uuid: "3552dd9467a749bbb1f161821fe54793"
format_stamp: "Formatted at 2026-08-12 02:36:16 on dist-test-slave-2w3w"
server_key: "8f95f878113d9935791bee9e1a61a870"
server_key_iv: "2843495802459045db5589ae8e712217"
server_key_version: "encryptionkey@0"
I20260812 02:36:16.188758 3305 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501898713699-14748-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501898713699-14748-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501898713699-14748-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:36:16.205134 3305 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:36:16.208526 3305 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:36:16.208750 3305 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:36:16.209393 3305 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:36:16.210261 3305 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:36:16.210311 3305 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:36:16.210381 3305 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:36:16.210421 3305 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:36:16.219728 3305 rpc_server.cc:307] RPC server started. Bound to: 127.14.103.3:41265
I20260812 02:36:16.219817 3456 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.103.3:41265 every 8 connection(s)
I20260812 02:36:16.220690 3305 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501898713699-14748-0/minicluster-data/ts-2/data/info.pb
I20260812 02:36:16.228631 14748 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbep4M4/build/debug/bin/kudu as pid 3305
I20260812 02:36:16.228797 14748 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501898713699-14748-0/minicluster-data/ts-2/wal/instance
I20260812 02:36:16.229133 14748 external_mini_cluster.cc:1428] Setting key a5bfd2523b17b31f5331c4b4304b825a
Aug 12 02:36:16 dist-test-slave-2w3w krb5kdc[2876](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502176, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.3@KRBTEST.COM for kudu/127.14.103.62@KRBTEST.COM
I20260812 02:36:16.234764 2975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:16.222716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.3:59523 (local address 127.14.103.62:36313)
0812 02:36:16.222858 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0812 02:36:16.222861 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 02:36:16.223774 (+ 913us) server_negotiation.cc:407] Connection header received
0812 02:36:16.225174 (+ 1400us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:36:16.225177 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:36:16.225240 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:36:16.225346 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:36:16.226536 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:16.227172 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:16.228333 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:16.228568 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:16.228686 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:36:16.231478 (+ 2792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:36:16.231482 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:36:16.231484 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:36:16.231508 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:36:16.232892 (+ 1384us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:36:16.233384 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:36:16.233387 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:36:16.233388 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:36:16.233486 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:36:16.233785 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:36:16.233787 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:36:16.233788 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:36:16.233968 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:36:16.234049 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0812 02:36:16.234526 (+ 477us) server_negotiation.cc:299] Negotiation successful
0812 02:36:16.234627 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260812 02:36:16.235682 3461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:16.223139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:36313 (local address 127.14.103.3:59523)
0812 02:36:16.223614 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0812 02:36:16.223658 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0812 02:36:16.224945 (+ 1287us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:36:16.225523 (+ 578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:36:16.225531 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:36:16.225899 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:36:16.226369 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:36:16.226393 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:36:16.227299 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:36:16.227303 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:36:16.228193 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:36:16.228202 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:36:16.228727 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:36:16.228732 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:36:16.228960 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:36:16.229592 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:36:16.229608 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:36:16.231298 (+ 1690us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:36:16.233004 (+ 1706us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:36:16.233009 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:36:16.233021 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:36:16.233295 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:36:16.233570 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:36:16.233572 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:36:16.233573 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:36:16.233706 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:36:16.234061 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:36:16.234066 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:36:16.234308 (+ 242us) client_negotiation.cc:769] Sending connection context
0812 02:36:16.234470 (+ 162us) client_negotiation.cc:240] Negotiation successful
0812 02:36:16.234734 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":126,"threads_started":1}
I20260812 02:36:16.236985 3459 heartbeater.cc:347] Connected to a master server at 127.14.103.62:36313
I20260812 02:36:16.237280 3459 heartbeater.cc:464] Registering TS with master...
I20260812 02:36:16.237813 3459 heartbeater.cc:511] Master 127.14.103.62:36313 requested a full tablet report, sending...
I20260812 02:36:16.238895 2927 ts_manager.cc:194] Registered new tserver with Master: 3552dd9467a749bbb1f161821fe54793 (127.14.103.3:41265)
I20260812 02:36:16.239540 2927 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.103.3@KRBTEST.COM'} at 127.14.103.3:59523
I20260812 02:36:16.243358 14748 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 02:36:16.253895 2975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:16.245563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47850 (local address 127.14.103.62:36313)
0812 02:36:16.245741 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0812 02:36:16.245747 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 02:36:16.246008 (+ 261us) server_negotiation.cc:407] Connection header received
0812 02:36:16.246174 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:36:16.246177 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:36:16.246228 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:36:16.246315 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:36:16.247185 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:16.247722 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:16.248575 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:16.248793 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:16.248901 (+ 108us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:36:16.250547 (+ 1646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:36:16.250551 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:36:16.250554 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:36:16.250585 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:36:16.252028 (+ 1443us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:36:16.252549 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:36:16.252558 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:36:16.252559 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:36:16.252634 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:36:16.252969 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:36:16.252972 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:36:16.252974 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:36:16.253190 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:36:16.253353 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0812 02:36:16.253634 (+ 281us) server_negotiation.cc:299] Negotiation successful
0812 02:36:16.253728 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260812 02:36:16.256918 2927 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47850:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 02:36:16.259927 2927 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 02:36:16.274770 3473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:16.268681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.2:40011 (local address 127.0.0.1:42238)
0812 02:36:16.269043 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0812 02:36:16.269106 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0812 02:36:16.269266 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:36:16.270121 (+ 855us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:36:16.270127 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:36:16.270149 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:36:16.270308 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:36:16.270316 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:36:16.271652 (+ 1336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:36:16.271659 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:36:16.272936 (+ 1277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:36:16.272946 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:36:16.274060 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:36:16.274064 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:36:16.274217 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:36:16.274301 (+ 84us) client_negotiation.cc:769] Sending connection context
0812 02:36:16.274407 (+ 106us) client_negotiation.cc:240] Negotiation successful
0812 02:36:16.274540 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"mutex_wait_us":24,"thread_start_us":87,"threads_started":1}
I20260812 02:36:16.274797 3474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:16.268821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42238 (local address 127.14.103.2:40011)
0812 02:36:16.269669 (+ 848us) server_negotiation.cc:206] Beginning negotiation
0812 02:36:16.269682 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0812 02:36:16.269707 (+ 25us) server_negotiation.cc:407] Connection header received
0812 02:36:16.269788 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:36:16.269794 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:36:16.269968 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:36:16.270130 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:36:16.270437 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:16.271517 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:16.273067 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:16.273927 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:16.274042 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:36:16.274205 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0812 02:36:16.274537 (+ 332us) server_negotiation.cc:299] Negotiation successful
0812 02:36:16.274640 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":735,"thread_start_us":151,"threads_started":1}
I20260812 02:36:16.275360 3476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:16.269390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.3:41265 (local address 127.0.0.1:44984)
0812 02:36:16.269616 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0812 02:36:16.269628 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 02:36:16.269704 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:36:16.270606 (+ 902us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:36:16.270609 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:36:16.270629 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:36:16.270720 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:36:16.270726 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:36:16.272122 (+ 1396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:36:16.272126 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:36:16.273209 (+ 1083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:36:16.273218 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:36:16.274308 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:36:16.274312 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:36:16.274413 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:36:16.274436 (+ 23us) client_negotiation.cc:769] Sending connection context
0812 02:36:16.274523 (+ 87us) client_negotiation.cc:240] Negotiation successful
0812 02:36:16.274599 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":107,"threads_started":1}
I20260812 02:36:16.275741 3475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:16.269074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.1:38823 (local address 127.0.0.1:59838)
0812 02:36:16.269332 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0812 02:36:16.269346 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 02:36:16.269424 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:36:16.271691 (+ 2267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:36:16.271694 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:36:16.271711 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:36:16.271799 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:36:16.271804 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:36:16.272892 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:36:16.272898 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:36:16.274098 (+ 1200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:36:16.274108 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:36:16.275318 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:36:16.275322 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:36:16.275410 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:36:16.275433 (+ 23us) client_negotiation.cc:769] Sending connection context
0812 02:36:16.275520 (+ 87us) client_negotiation.cc:240] Negotiation successful
0812 02:36:16.275583 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"mutex_wait_us":3,"thread_start_us":96,"threads_started":1}
I20260812 02:36:16.276062 3478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:16.269829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44984 (local address 127.14.103.3:41265)
0812 02:36:16.270156 (+ 327us) server_negotiation.cc:206] Beginning negotiation
0812 02:36:16.270168 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 02:36:16.270190 (+ 22us) server_negotiation.cc:407] Connection header received
0812 02:36:16.270266 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:36:16.270271 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:36:16.270456 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:36:16.270613 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:36:16.270849 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:16.271960 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:16.273346 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:16.274192 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:16.274301 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:36:16.274445 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0812 02:36:16.275821 (+ 1376us) server_negotiation.cc:299] Negotiation successful
0812 02:36:16.275926 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":236,"thread_start_us":159,"threads_started":1}
I20260812 02:36:16.276489 3477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:16.269492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59838 (local address 127.14.103.1:38823)
0812 02:36:16.271255 (+ 1763us) server_negotiation.cc:206] Beginning negotiation
0812 02:36:16.271261 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 02:36:16.271284 (+ 23us) server_negotiation.cc:407] Connection header received
0812 02:36:16.271359 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:36:16.271364 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:36:16.271541 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:36:16.271923 (+ 382us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:36:16.272062 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:16.272772 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:16.274237 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:16.275174 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:16.275309 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:36:16.275453 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0812 02:36:16.275557 (+ 104us) server_negotiation.cc:299] Negotiation successful
0812 02:36:16.275658 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":840,"thread_start_us":95,"threads_started":1}
I20260812 02:36:16.278020 3227 tablet_service.cc:1467] Processing CreateTablet for tablet 7f64ace71b0e4037a4596a8dd2bdd40f (DEFAULT_TABLE table=test-table [id=9dcc84c073374d74939647a3379ce94c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:36:16.278371 3377 tablet_service.cc:1467] Processing CreateTablet for tablet 7f64ace71b0e4037a4596a8dd2bdd40f (DEFAULT_TABLE table=test-table [id=9dcc84c073374d74939647a3379ce94c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:36:16.279211 3227 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f64ace71b0e4037a4596a8dd2bdd40f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:36:16.279425 3377 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f64ace71b0e4037a4596a8dd2bdd40f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:36:16.281571 3071 tablet_service.cc:1467] Processing CreateTablet for tablet 7f64ace71b0e4037a4596a8dd2bdd40f (DEFAULT_TABLE table=test-table [id=9dcc84c073374d74939647a3379ce94c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:36:16.282617 3071 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f64ace71b0e4037a4596a8dd2bdd40f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:36:16.285467 3479 tablet_bootstrap.cc:492] T 7f64ace71b0e4037a4596a8dd2bdd40f P 3552dd9467a749bbb1f161821fe54793: Bootstrap starting.
I20260812 02:36:16.286798 3480 tablet_bootstrap.cc:492] T 7f64ace71b0e4037a4596a8dd2bdd40f P 2207d0e8f81d4c338903fe9c291ba2b1: Bootstrap starting.
I20260812 02:36:16.288189 3479 tablet_bootstrap.cc:654] T 7f64ace71b0e4037a4596a8dd2bdd40f P 3552dd9467a749bbb1f161821fe54793: Neither blocks nor log segments found. Creating new log.
I20260812 02:36:16.289500 3480 tablet_bootstrap.cc:654] T 7f64ace71b0e4037a4596a8dd2bdd40f P 2207d0e8f81d4c338903fe9c291ba2b1: Neither blocks nor log segments found. Creating new log.
I20260812 02:36:16.289815 3481 tablet_bootstrap.cc:492] T 7f64ace71b0e4037a4596a8dd2bdd40f P 947908d10e4d423eb3d4fc5fd99a8b2f: Bootstrap starting.
I20260812 02:36:16.289806 3479 log.cc:824] T 7f64ace71b0e4037a4596a8dd2bdd40f P 3552dd9467a749bbb1f161821fe54793: Log is configured to *not* fsync() on all Append() calls
I20260812 02:36:16.290427 3480 log.cc:824] T 7f64ace71b0e4037a4596a8dd2bdd40f P 2207d0e8f81d4c338903fe9c291ba2b1: Log is configured to *not* fsync() on all Append() calls
I20260812 02:36:16.292241 3479 tablet_bootstrap.cc:492] T 7f64ace71b0e4037a4596a8dd2bdd40f P 3552dd9467a749bbb1f161821fe54793: No bootstrap required, opened a new log
I20260812 02:36:16.292250 3481 tablet_bootstrap.cc:654] T 7f64ace71b0e4037a4596a8dd2bdd40f P 947908d10e4d423eb3d4fc5fd99a8b2f: Neither blocks nor log segments found. Creating new log.
I20260812 02:36:16.292464 3479 ts_tablet_manager.cc:1397] T 7f64ace71b0e4037a4596a8dd2bdd40f P 3552dd9467a749bbb1f161821fe54793: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260812 02:36:16.292567 3480 tablet_bootstrap.cc:492] T 7f64ace71b0e4037a4596a8dd2bdd40f P 2207d0e8f81d4c338903fe9c291ba2b1: No bootstrap required, opened a new log
I20260812 02:36:16.292752 3480 ts_tablet_manager.cc:1397] T 7f64ace71b0e4037a4596a8dd2bdd40f P 2207d0e8f81d4c338903fe9c291ba2b1: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260812 02:36:16.293044 3481 log.cc:824] T 7f64ace71b0e4037a4596a8dd2bdd40f P 947908d10e4d423eb3d4fc5fd99a8b2f: Log is configured to *not* fsync() on all Append() calls
I20260812 02:36:16.295172 3481 tablet_bootstrap.cc:492] T 7f64ace71b0e4037a4596a8dd2bdd40f P 947908d10e4d423eb3d4fc5fd99a8b2f: No bootstrap required, opened a new log
I20260812 02:36:16.295392 3481 ts_tablet_manager.cc:1397] T 7f64ace71b0e4037a4596a8dd2bdd40f P 947908d10e4d423eb3d4fc5fd99a8b2f: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.004s
I20260812 02:36:16.296283 3479 raft_consensus.cc:348] T 7f64ace71b0e4037a4596a8dd2bdd40f P 3552dd9467a749bbb1f161821fe54793 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2207d0e8f81d4c338903fe9c291ba2b1" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 40011 } } peers { permanent_uuid: "3552dd9467a749bbb1f161821fe54793" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 41265 } } peers { permanent_uuid: "947908d10e4d423eb3d4fc5fd99a8b2f" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 38823 } }
I20260812 02:36:16.296552 3479 raft_consensus.cc:374] T 7f64ace71b0e4037a4596a8dd2bdd40f P 3552dd9467a749bbb1f161821fe54793 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:36:16.296634 3479 raft_consensus.cc:729] T 7f64ace71b0e4037a4596a8dd2bdd40f P 3552dd9467a749bbb1f161821fe54793 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3552dd9467a749bbb1f161821fe54793, State: Initialized, Role: FOLLOWER
I20260812 02:36:16.296576 3480 raft_consensus.cc:348] T 7f64ace71b0e4037a4596a8dd2bdd40f P 2207d0e8f81d4c338903fe9c291ba2b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2207d0e8f81d4c338903fe9c291ba2b1" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 40011 } } peers { permanent_uuid: "3552dd9467a749bbb1f161821fe54793" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 41265 } } peers { permanent_uuid: "947908d10e4d423eb3d4fc5fd99a8b2f" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 38823 } }
I20260812 02:36:16.296836 3480 raft_consensus.cc:374] T 7f64ace71b0e4037a4596a8dd2bdd40f P 2207d0e8f81d4c338903fe9c291ba2b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:36:16.296944 3480 raft_consensus.cc:729] T 7f64ace71b0e4037a4596a8dd2bdd40f P 2207d0e8f81d4c338903fe9c291ba2b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2207d0e8f81d4c338903fe9c291ba2b1, State: Initialized, Role: FOLLOWER
I20260812 02:36:16.297165 3479 consensus_queue.cc:260] T 7f64ace71b0e4037a4596a8dd2bdd40f P 3552dd9467a749bbb1f161821fe54793 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2207d0e8f81d4c338903fe9c291ba2b1" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 40011 } } peers { permanent_uuid: "3552dd9467a749bbb1f161821fe54793" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 41265 } } peers { permanent_uuid: "947908d10e4d423eb3d4fc5fd99a8b2f" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 38823 } }
I20260812 02:36:16.297472 3480 consensus_queue.cc:260] T 7f64ace71b0e4037a4596a8dd2bdd40f P 2207d0e8f81d4c338903fe9c291ba2b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2207d0e8f81d4c338903fe9c291ba2b1" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 40011 } } peers { permanent_uuid: "3552dd9467a749bbb1f161821fe54793" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 41265 } } peers { permanent_uuid: "947908d10e4d423eb3d4fc5fd99a8b2f" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 38823 } }
I20260812 02:36:16.297851 3459 heartbeater.cc:503] Master 127.14.103.62:36313 was elected leader, sending a full tablet report...
I20260812 02:36:16.298219 3479 ts_tablet_manager.cc:1428] T 7f64ace71b0e4037a4596a8dd2bdd40f P 3552dd9467a749bbb1f161821fe54793: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260812 02:36:16.298426 3480 ts_tablet_manager.cc:1428] T 7f64ace71b0e4037a4596a8dd2bdd40f P 2207d0e8f81d4c338903fe9c291ba2b1: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260812 02:36:16.298781 3300 heartbeater.cc:503] Master 127.14.103.62:36313 was elected leader, sending a full tablet report...
I20260812 02:36:16.299865 3481 raft_consensus.cc:348] T 7f64ace71b0e4037a4596a8dd2bdd40f P 947908d10e4d423eb3d4fc5fd99a8b2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2207d0e8f81d4c338903fe9c291ba2b1" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 40011 } } peers { permanent_uuid: "3552dd9467a749bbb1f161821fe54793" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 41265 } } peers { permanent_uuid: "947908d10e4d423eb3d4fc5fd99a8b2f" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 38823 } }
I20260812 02:36:16.300141 3481 raft_consensus.cc:374] T 7f64ace71b0e4037a4596a8dd2bdd40f P 947908d10e4d423eb3d4fc5fd99a8b2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:36:16.300244 3481 raft_consensus.cc:729] T 7f64ace71b0e4037a4596a8dd2bdd40f P 947908d10e4d423eb3d4fc5fd99a8b2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 947908d10e4d423eb3d4fc5fd99a8b2f, State: Initialized, Role: FOLLOWER
I20260812 02:36:16.300688 3481 consensus_queue.cc:260] T 7f64ace71b0e4037a4596a8dd2bdd40f P 947908d10e4d423eb3d4fc5fd99a8b2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2207d0e8f81d4c338903fe9c291ba2b1" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 40011 } } peers { permanent_uuid: "3552dd9467a749bbb1f161821fe54793" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 41265 } } peers { permanent_uuid: "947908d10e4d423eb3d4fc5fd99a8b2f" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 38823 } }
I20260812 02:36:16.301672 3481 ts_tablet_manager.cc:1428] T 7f64ace71b0e4037a4596a8dd2bdd40f P 947908d10e4d423eb3d4fc5fd99a8b2f: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260812 02:36:16.302030 3147 heartbeater.cc:503] Master 127.14.103.62:36313 was elected leader, sending a full tablet report...
I20260812 02:36:16.323364 3488 raft_consensus.cc:482] T 7f64ace71b0e4037a4596a8dd2bdd40f P 3552dd9467a749bbb1f161821fe54793 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 02:36:16.323614 3488 raft_consensus.cc:504] T 7f64ace71b0e4037a4596a8dd2bdd40f P 3552dd9467a749bbb1f161821fe54793 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2207d0e8f81d4c338903fe9c291ba2b1" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 40011 } } peers { permanent_uuid: "3552dd9467a749bbb1f161821fe54793" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 41265 } } peers { permanent_uuid: "947908d10e4d423eb3d4fc5fd99a8b2f" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 38823 } }
I20260812 02:36:16.324990 3488 leader_election.cc:288] T 7f64ace71b0e4037a4596a8dd2bdd40f P 3552dd9467a749bbb1f161821fe54793 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2207d0e8f81d4c338903fe9c291ba2b1 (127.14.103.2:40011), 947908d10e4d423eb3d4fc5fd99a8b2f (127.14.103.1:38823)
I20260812 02:36:16.329351 3461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:16.325233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.2:40011 (local address 127.14.103.3:51323)
0812 02:36:16.325499 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0812 02:36:16.325517 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 02:36:16.325633 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:36:16.325950 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:36:16.325953 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:36:16.326001 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:36:16.326151 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:36:16.326159 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:36:16.327107 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:36:16.327110 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:36:16.328096 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:36:16.328106 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:36:16.328857 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:36:16.328861 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:36:16.329020 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:36:16.329052 (+ 32us) client_negotiation.cc:769] Sending connection context
0812 02:36:16.329130 (+ 78us) client_negotiation.cc:240] Negotiation successful
0812 02:36:16.329217 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179}
I20260812 02:36:16.329419 3474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:16.325363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.3:51323 (local address 127.14.103.2:40011)
0812 02:36:16.325500 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0812 02:36:16.325507 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 02:36:16.325616 (+ 109us) server_negotiation.cc:407] Connection header received
0812 02:36:16.325767 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:36:16.325770 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:36:16.325831 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:36:16.325925 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:36:16.326318 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:16.326932 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:16.328240 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:16.328715 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:16.328817 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:36:16.328858 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0812 02:36:16.329196 (+ 338us) server_negotiation.cc:299] Negotiation successful
0812 02:36:16.329278 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260812 02:36:16.329550 3491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:16.325324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.1:38823 (local address 127.14.103.3:42281)
0812 02:36:16.325909 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0812 02:36:16.325925 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 02:36:16.326028 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:36:16.326318 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:36:16.326321 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:36:16.326339 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:36:16.326455 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:36:16.326461 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:36:16.327423 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:36:16.327426 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:36:16.328248 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:36:16.328258 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:36:16.329083 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:36:16.329086 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:36:16.329246 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:36:16.329268 (+ 22us) client_negotiation.cc:769] Sending connection context
0812 02:36:16.329351 (+ 83us) client_negotiation.cc:240] Negotiation successful
0812 02:36:16.329415 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":166,"threads_started":1}
I20260812 02:36:16.329648 3477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:16.325469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.3:42281 (local address 127.14.103.1:38823)
0812 02:36:16.325642 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0812 02:36:16.325647 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:36:16.326026 (+ 379us) server_negotiation.cc:407] Connection header received
0812 02:36:16.326142 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:36:16.326145 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:36:16.326208 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:36:16.326298 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:36:16.326570 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:16.327308 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:16.328399 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:16.328930 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:16.329079 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:36:16.329148 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0812 02:36:16.329392 (+ 244us) server_negotiation.cc:299] Negotiation successful
0812 02:36:16.329494 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260812 02:36:16.330261 3095 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f64ace71b0e4037a4596a8dd2bdd40f" candidate_uuid: "3552dd9467a749bbb1f161821fe54793" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "947908d10e4d423eb3d4fc5fd99a8b2f" is_pre_election: true
I20260812 02:36:16.330261 3250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f64ace71b0e4037a4596a8dd2bdd40f" candidate_uuid: "3552dd9467a749bbb1f161821fe54793" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2207d0e8f81d4c338903fe9c291ba2b1" is_pre_election: true
I20260812 02:36:16.330579 3250 raft_consensus.cc:2454] T 7f64ace71b0e4037a4596a8dd2bdd40f P 2207d0e8f81d4c338903fe9c291ba2b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3552dd9467a749bbb1f161821fe54793 in term 0.
I20260812 02:36:16.330593 3095 raft_consensus.cc:2454] T 7f64ace71b0e4037a4596a8dd2bdd40f P 947908d10e4d423eb3d4fc5fd99a8b2f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3552dd9467a749bbb1f161821fe54793 in term 0.
I20260812 02:36:16.331126 3328 leader_election.cc:302] T 7f64ace71b0e4037a4596a8dd2bdd40f P 3552dd9467a749bbb1f161821fe54793 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2207d0e8f81d4c338903fe9c291ba2b1, 3552dd9467a749bbb1f161821fe54793, 947908d10e4d423eb3d4fc5fd99a8b2f; no voters:
I20260812 02:36:16.331436 3488 raft_consensus.cc:2781] T 7f64ace71b0e4037a4596a8dd2bdd40f P 3552dd9467a749bbb1f161821fe54793 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 02:36:16.331526 3488 raft_consensus.cc:482] T 7f64ace71b0e4037a4596a8dd2bdd40f P 3552dd9467a749bbb1f161821fe54793 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 02:36:16.331558 3488 raft_consensus.cc:3037] T 7f64ace71b0e4037a4596a8dd2bdd40f P 3552dd9467a749bbb1f161821fe54793 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:36:16.332669 3488 raft_consensus.cc:504] T 7f64ace71b0e4037a4596a8dd2bdd40f P 3552dd9467a749bbb1f161821fe54793 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2207d0e8f81d4c338903fe9c291ba2b1" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 40011 } } peers { permanent_uuid: "3552dd9467a749bbb1f161821fe54793" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 41265 } } peers { permanent_uuid: "947908d10e4d423eb3d4fc5fd99a8b2f" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 38823 } }
I20260812 02:36:16.333096 3488 leader_election.cc:288] T 7f64ace71b0e4037a4596a8dd2bdd40f P 3552dd9467a749bbb1f161821fe54793 [CANDIDATE]: Term 1 election: Requested vote from peers 2207d0e8f81d4c338903fe9c291ba2b1 (127.14.103.2:40011), 947908d10e4d423eb3d4fc5fd99a8b2f (127.14.103.1:38823)
I20260812 02:36:16.333516 3250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f64ace71b0e4037a4596a8dd2bdd40f" candidate_uuid: "3552dd9467a749bbb1f161821fe54793" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2207d0e8f81d4c338903fe9c291ba2b1"
I20260812 02:36:16.333578 3095 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f64ace71b0e4037a4596a8dd2bdd40f" candidate_uuid: "3552dd9467a749bbb1f161821fe54793" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "947908d10e4d423eb3d4fc5fd99a8b2f"
I20260812 02:36:16.333668 3250 raft_consensus.cc:3037] T 7f64ace71b0e4037a4596a8dd2bdd40f P 2207d0e8f81d4c338903fe9c291ba2b1 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:36:16.333734 3095 raft_consensus.cc:3037] T 7f64ace71b0e4037a4596a8dd2bdd40f P 947908d10e4d423eb3d4fc5fd99a8b2f [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:36:16.335103 3250 raft_consensus.cc:2454] T 7f64ace71b0e4037a4596a8dd2bdd40f P 2207d0e8f81d4c338903fe9c291ba2b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3552dd9467a749bbb1f161821fe54793 in term 1.
I20260812 02:36:16.335203 3095 raft_consensus.cc:2454] T 7f64ace71b0e4037a4596a8dd2bdd40f P 947908d10e4d423eb3d4fc5fd99a8b2f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3552dd9467a749bbb1f161821fe54793 in term 1.
I20260812 02:36:16.335443 3326 leader_election.cc:302] T 7f64ace71b0e4037a4596a8dd2bdd40f P 3552dd9467a749bbb1f161821fe54793 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2207d0e8f81d4c338903fe9c291ba2b1, 3552dd9467a749bbb1f161821fe54793; no voters:
I20260812 02:36:16.335613 3488 raft_consensus.cc:2781] T 7f64ace71b0e4037a4596a8dd2bdd40f P 3552dd9467a749bbb1f161821fe54793 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:36:16.335851 3488 raft_consensus.cc:686] T 7f64ace71b0e4037a4596a8dd2bdd40f P 3552dd9467a749bbb1f161821fe54793 [term 1 LEADER]: Becoming Leader. State: Replica: 3552dd9467a749bbb1f161821fe54793, State: Running, Role: LEADER
I20260812 02:36:16.336200 3488 consensus_queue.cc:237] T 7f64ace71b0e4037a4596a8dd2bdd40f P 3552dd9467a749bbb1f161821fe54793 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2207d0e8f81d4c338903fe9c291ba2b1" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 40011 } } peers { permanent_uuid: "3552dd9467a749bbb1f161821fe54793" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 41265 } } peers { permanent_uuid: "947908d10e4d423eb3d4fc5fd99a8b2f" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 38823 } }
I20260812 02:36:16.339524 2924 catalog_manager.cc:5495] T 7f64ace71b0e4037a4596a8dd2bdd40f P 3552dd9467a749bbb1f161821fe54793 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3552dd9467a749bbb1f161821fe54793 (127.14.103.3). New cstate: current_term: 1 leader_uuid: "3552dd9467a749bbb1f161821fe54793" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2207d0e8f81d4c338903fe9c291ba2b1" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 40011 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3552dd9467a749bbb1f161821fe54793" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 41265 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "947908d10e4d423eb3d4fc5fd99a8b2f" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 38823 } health_report { overall_health: UNKNOWN } } }
I20260812 02:36:16.359022 3478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:16.355311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44994 (local address 127.14.103.3:41265)
0812 02:36:16.355452 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0812 02:36:16.355456 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:36:16.355471 (+ 15us) server_negotiation.cc:407] Connection header received
0812 02:36:16.355598 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:36:16.355601 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:36:16.355655 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:36:16.355771 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 02:36:16.356033 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:16.356552 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:16.357428 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:16.357649 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:16.357757 (+ 108us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:36:16.358099 (+ 342us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 02:36:16.358511 (+ 412us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 02:36:16.358673 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0812 02:36:16.358766 (+ 93us) server_negotiation.cc:299] Negotiation successful
0812 02:36:16.358805 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260812 02:36:16.365233 3095 raft_consensus.cc:1261] T 7f64ace71b0e4037a4596a8dd2bdd40f P 947908d10e4d423eb3d4fc5fd99a8b2f [term 1 FOLLOWER]: Refusing update from remote peer 3552dd9467a749bbb1f161821fe54793: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:36:16.365232 3250 raft_consensus.cc:1261] T 7f64ace71b0e4037a4596a8dd2bdd40f P 2207d0e8f81d4c338903fe9c291ba2b1 [term 1 FOLLOWER]: Refusing update from remote peer 3552dd9467a749bbb1f161821fe54793: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:36:16.366001 3488 consensus_queue.cc:1035] T 7f64ace71b0e4037a4596a8dd2bdd40f P 3552dd9467a749bbb1f161821fe54793 [LEADER]: Connected to new peer: Peer: permanent_uuid: "947908d10e4d423eb3d4fc5fd99a8b2f" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 38823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:36:16.366374 3492 consensus_queue.cc:1035] T 7f64ace71b0e4037a4596a8dd2bdd40f P 3552dd9467a749bbb1f161821fe54793 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2207d0e8f81d4c338903fe9c291ba2b1" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 40011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:36:16.374615 3498 mvcc.cc:205] Tried to move back new op lower bound from 7317512914384375808 to 7317512914275803136. Current Snapshot: MvccSnapshot[applied={T|T < 7317512914384375808}]
I20260812 02:36:16.383531 2927 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47850:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 02:36:16.383731 2927 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47850:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 02:36:16.386420 2927 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ea3e7b556e00409fb52175828a07669a: Sending DeleteTablet for 3 replicas of tablet 7f64ace71b0e4037a4596a8dd2bdd40f
I20260812 02:36:16.387332 3377 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f64ace71b0e4037a4596a8dd2bdd40f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:36:16 UTC) from {username='kudu', principal='kudu/127.14.103.62@KRBTEST.COM'} at 127.0.0.1:44984
I20260812 02:36:16.387343 3227 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f64ace71b0e4037a4596a8dd2bdd40f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:36:16 UTC) from {username='kudu', principal='kudu/127.14.103.62@KRBTEST.COM'} at 127.0.0.1:42238
I20260812 02:36:16.387526 3071 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f64ace71b0e4037a4596a8dd2bdd40f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:36:16 UTC) from {username='kudu', principal='kudu/127.14.103.62@KRBTEST.COM'} at 127.0.0.1:59838
I20260812 02:36:16.387909 3506 tablet_replica.cc:330] T 7f64ace71b0e4037a4596a8dd2bdd40f P 3552dd9467a749bbb1f161821fe54793: stopping tablet replica
I20260812 02:36:16.388263 3506 raft_consensus.cc:2229] T 7f64ace71b0e4037a4596a8dd2bdd40f P 3552dd9467a749bbb1f161821fe54793 [term 1 LEADER]: Raft consensus shutting down.
I20260812 02:36:16.388696 14748 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbep4M4/build/debug/bin/kudu with pid 3002
I20260812 02:36:16.388795 3506 raft_consensus.cc:2258] T 7f64ace71b0e4037a4596a8dd2bdd40f P 3552dd9467a749bbb1f161821fe54793 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:36:16.391084 3506 ts_tablet_manager.cc:1905] T 7f64ace71b0e4037a4596a8dd2bdd40f P 3552dd9467a749bbb1f161821fe54793: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:36:16.394280 3506 ts_tablet_manager.cc:1918] T 7f64ace71b0e4037a4596a8dd2bdd40f P 3552dd9467a749bbb1f161821fe54793: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 02:36:16.394414 3506 log.cc:1196] T 7f64ace71b0e4037a4596a8dd2bdd40f P 3552dd9467a749bbb1f161821fe54793: Deleting WAL directory at /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501898713699-14748-0/minicluster-data/ts-2/wal/wals/7f64ace71b0e4037a4596a8dd2bdd40f
I20260812 02:36:16.394779 3506 ts_tablet_manager.cc:1939] T 7f64ace71b0e4037a4596a8dd2bdd40f P 3552dd9467a749bbb1f161821fe54793: Deleting consensus metadata
I20260812 02:36:16.395227 3507 tablet_replica.cc:330] T 7f64ace71b0e4037a4596a8dd2bdd40f P 2207d0e8f81d4c338903fe9c291ba2b1: stopping tablet replica
I20260812 02:36:16.395735 3507 raft_consensus.cc:2229] T 7f64ace71b0e4037a4596a8dd2bdd40f P 2207d0e8f81d4c338903fe9c291ba2b1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 02:36:16.396149 3507 raft_consensus.cc:2258] T 7f64ace71b0e4037a4596a8dd2bdd40f P 2207d0e8f81d4c338903fe9c291ba2b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:36:16.396569 2909 catalog_manager.cc:4841] TS 3552dd9467a749bbb1f161821fe54793 (127.14.103.3:41265): tablet 7f64ace71b0e4037a4596a8dd2bdd40f (table test-table [id=9dcc84c073374d74939647a3379ce94c]) successfully deleted
I20260812 02:36:16.397727 3507 ts_tablet_manager.cc:1905] T 7f64ace71b0e4037a4596a8dd2bdd40f P 2207d0e8f81d4c338903fe9c291ba2b1: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260812 02:36:16.399818 2911 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.103.1:38823 (error 108)
I20260812 02:36:16.400354 3507 ts_tablet_manager.cc:1918] T 7f64ace71b0e4037a4596a8dd2bdd40f P 2207d0e8f81d4c338903fe9c291ba2b1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 02:36:16.400557 3507 log.cc:1196] T 7f64ace71b0e4037a4596a8dd2bdd40f P 2207d0e8f81d4c338903fe9c291ba2b1: Deleting WAL directory at /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786501898713699-14748-0/minicluster-data/ts-1/wal/wals/7f64ace71b0e4037a4596a8dd2bdd40f
I20260812 02:36:16.400965 3507 ts_tablet_manager.cc:1939] T 7f64ace71b0e4037a4596a8dd2bdd40f P 2207d0e8f81d4c338903fe9c291ba2b1: Deleting consensus metadata
I20260812 02:36:16.401026 3475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:16.400642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.1:38823 (local address 127.0.0.1:59844)
0812 02:36:16.400825 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0812 02:36:16.400917 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.1:38823: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260812 02:36:16.401050 14748 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbep4M4/build/debug/bin/kudu with pid 3155
W20260812 02:36:16.401301 2911 catalog_manager.cc:4584] TS 947908d10e4d423eb3d4fc5fd99a8b2f (127.14.103.1:38823): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7f64ace71b0e4037a4596a8dd2bdd40f: Network error: Client connection negotiation failed: client connection to 127.14.103.1:38823: connect: Connection refused (error 111)
I20260812 02:36:16.415477 14748 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbep4M4/build/debug/bin/kudu with pid 3305
I20260812 02:36:16.415791 3475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:16.415471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.2:40011 (local address 127.0.0.1:42254)
0812 02:36:16.415609 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0812 02:36:16.415677 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.2:40011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260812 02:36:16.423383 14748 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbep4M4/build/debug/bin/kudu with pid 2890
2026-08-12T02:36:16Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2456 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-08-12T02:36:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T02:36:16Z Disabled control of system clock
I20260812 02:36:16.458443 14748 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbep4M4/build/debug/bin/kudu
/tmp/dist-test-taskbep4M4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786501898713699-14748-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786501898713699-14748-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786501898713699-14748-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786501898713699-14748-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.103.62:33947
--webserver_interface=127.14.103.62
--webserver_port=0
--builtin_ntp_servers=127.14.103.20:42751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.103.62:33947
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260812 02:36:16.572857 3515 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:36:16.573155 3515 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:36:16.573196 3515 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:36:16.577536 3515 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:36:16.577613 3515 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:36:16.577630 3515 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:36:16.577651 3515 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:36:16.577665 3515 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 02:36:16.582643 3515 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.103.20:42751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786501898713699-14748-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786501898713699-14748-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.103.62:33947
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.103.62:33947
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786501898713699-14748-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.103.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3515
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786501898713699-14748-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:36:16.583841 3515 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:36:16.585263 3515 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260812 02:36:16.594525 3515 server_base.cc:1034] running on GCE node
W20260812 02:36:16.594429 3523 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:36:16.594424 3520 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:36:16.594429 3521 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:36:16.595451 3515 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:36:16.596662 3515 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:36:16.600427 3515 hybrid_clock.cc:648] HybridClock initialized: now 1786502176600383 us; error 48 us; skew 500 ppm
I20260812 02:36:16.602676 3515 webserver.cc:460] Webserver started at http://127.14.103.62:41835/ using document root <none> and password file <none>
I20260812 02:36:16.603384 3515 fs_manager.cc:359] Metadata directory not provided
I20260812 02:36:16.603494 3515 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:36:16.603744 3515 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:36:16.606042 3515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786501898713699-14748-0/minicluster-data/master-0/data/instance:
uuid: "ef7ffb878cff42f2ac0d403ad8efbf07"
format_stamp: "Formatted at 2026-08-12 02:36:16 on dist-test-slave-2w3w"
server_key: "4adc899548f1ce3ae3a18e4f6f0ec8e9"
server_key_iv: "07f09a1d76f41ece285de5dac4429a64"
server_key_version: "encryptionkey@0"
I20260812 02:36:16.606652 3515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786501898713699-14748-0/minicluster-data/master-0/wal/instance:
uuid: "ef7ffb878cff42f2ac0d403ad8efbf07"
format_stamp: "Formatted at 2026-08-12 02:36:16 on dist-test-slave-2w3w"
server_key: "4adc899548f1ce3ae3a18e4f6f0ec8e9"
server_key_iv: "07f09a1d76f41ece285de5dac4429a64"
server_key_version: "encryptionkey@0"
I20260812 02:36:16.611088 3515 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260812 02:36:16.613884 3530 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:36:16.615419 3515 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260812 02:36:16.615578 3515 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786501898713699-14748-0/minicluster-data/master-0/data,/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786501898713699-14748-0/minicluster-data/master-0/wal
uuid: "ef7ffb878cff42f2ac0d403ad8efbf07"
format_stamp: "Formatted at 2026-08-12 02:36:16 on dist-test-slave-2w3w"
server_key: "4adc899548f1ce3ae3a18e4f6f0ec8e9"
server_key_iv: "07f09a1d76f41ece285de5dac4429a64"
server_key_version: "encryptionkey@0"
I20260812 02:36:16.615712 3515 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786501898713699-14748-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786501898713699-14748-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786501898713699-14748-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:36:16.639281 3515 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:36:16.640049 3515 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:36:16.640267 3515 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:36:16.647513 3515 rpc_server.cc:307] RPC server started. Bound to: 127.14.103.62:33947
I20260812 02:36:16.647534 3596 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.103.62:33947 every 8 connection(s)
I20260812 02:36:16.648483 3515 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786501898713699-14748-0/minicluster-data/master-0/data/info.pb
I20260812 02:36:16.651695 3597 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:36:16.655421 14748 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbep4M4/build/debug/bin/kudu as pid 3515
I20260812 02:36:16.655586 14748 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786501898713699-14748-0/minicluster-data/master-0/wal/instance
I20260812 02:36:16.655872 14748 external_mini_cluster.cc:1428] Setting key 60f6a3bf62dbe410c98ba4654524e2c3
I20260812 02:36:16.657635 3597 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef7ffb878cff42f2ac0d403ad8efbf07: Bootstrap starting.
I20260812 02:36:16.660092 3597 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ef7ffb878cff42f2ac0d403ad8efbf07: Neither blocks nor log segments found. Creating new log.
I20260812 02:36:16.660866 3597 log.cc:824] T 00000000000000000000000000000000 P ef7ffb878cff42f2ac0d403ad8efbf07: Log is configured to *not* fsync() on all Append() calls
I20260812 02:36:16.663201 3597 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef7ffb878cff42f2ac0d403ad8efbf07: No bootstrap required, opened a new log
I20260812 02:36:16.665696 3600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:16.657418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52256 (local address 127.14.103.62:33947)
0812 02:36:16.658042 (+ 624us) server_negotiation.cc:206] Beginning negotiation
0812 02:36:16.658058 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0812 02:36:16.658094 (+ 36us) server_negotiation.cc:407] Connection header received
0812 02:36:16.658751 (+ 657us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:36:16.658766 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:36:16.659221 (+ 455us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:36:16.659493 (+ 272us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:36:16.659888 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:16.660940 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:16.662133 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:16.662601 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:16.662729 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:36:16.663849 (+ 1120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:36:16.663854 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:36:16.663867 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:36:16.664266 (+ 399us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:36:16.664325 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:36:16.664345 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:36:16.664443 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0812 02:36:16.664614 (+ 171us) server_negotiation.cc:299] Negotiation successful
0812 02:36:16.664917 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":134,"threads_started":1}
I20260812 02:36:16.667471 3597 raft_consensus.cc:348] T 00000000000000000000000000000000 P ef7ffb878cff42f2ac0d403ad8efbf07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef7ffb878cff42f2ac0d403ad8efbf07" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 33947 } }
I20260812 02:36:16.667723 3597 raft_consensus.cc:374] T 00000000000000000000000000000000 P ef7ffb878cff42f2ac0d403ad8efbf07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:36:16.667810 3597 raft_consensus.cc:729] T 00000000000000000000000000000000 P ef7ffb878cff42f2ac0d403ad8efbf07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef7ffb878cff42f2ac0d403ad8efbf07, State: Initialized, Role: FOLLOWER
I20260812 02:36:16.668334 3597 consensus_queue.cc:260] T 00000000000000000000000000000000 P ef7ffb878cff42f2ac0d403ad8efbf07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef7ffb878cff42f2ac0d403ad8efbf07" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 33947 } }
I20260812 02:36:16.668500 3597 raft_consensus.cc:388] T 00000000000000000000000000000000 P ef7ffb878cff42f2ac0d403ad8efbf07 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 02:36:16.668566 3597 raft_consensus.cc:482] T 00000000000000000000000000000000 P ef7ffb878cff42f2ac0d403ad8efbf07 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 02:36:16.668664 3597 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ef7ffb878cff42f2ac0d403ad8efbf07 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:36:16.669776 3597 raft_consensus.cc:504] T 00000000000000000000000000000000 P ef7ffb878cff42f2ac0d403ad8efbf07 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef7ffb878cff42f2ac0d403ad8efbf07" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 33947 } }
I20260812 02:36:16.670135 3597 leader_election.cc:302] T 00000000000000000000000000000000 P ef7ffb878cff42f2ac0d403ad8efbf07 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ef7ffb878cff42f2ac0d403ad8efbf07; no voters:
I20260812 02:36:16.670459 3597 leader_election.cc:288] T 00000000000000000000000000000000 P ef7ffb878cff42f2ac0d403ad8efbf07 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 02:36:16.670586 3602 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ef7ffb878cff42f2ac0d403ad8efbf07 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:36:16.670866 3602 raft_consensus.cc:686] T 00000000000000000000000000000000 P ef7ffb878cff42f2ac0d403ad8efbf07 [term 1 LEADER]: Becoming Leader. State: Replica: ef7ffb878cff42f2ac0d403ad8efbf07, State: Running, Role: LEADER
I20260812 02:36:16.671257 3602 consensus_queue.cc:237] T 00000000000000000000000000000000 P ef7ffb878cff42f2ac0d403ad8efbf07 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef7ffb878cff42f2ac0d403ad8efbf07" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 33947 } }
I20260812 02:36:16.671897 3597 sys_catalog.cc:564] T 00000000000000000000000000000000 P ef7ffb878cff42f2ac0d403ad8efbf07 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:36:16.673130 3604 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef7ffb878cff42f2ac0d403ad8efbf07 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ef7ffb878cff42f2ac0d403ad8efbf07. Latest consensus state: current_term: 1 leader_uuid: "ef7ffb878cff42f2ac0d403ad8efbf07" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef7ffb878cff42f2ac0d403ad8efbf07" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 33947 } } }
I20260812 02:36:16.673267 3604 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef7ffb878cff42f2ac0d403ad8efbf07 [sys.catalog]: This master's current role is: LEADER
I20260812 02:36:16.673614 3603 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef7ffb878cff42f2ac0d403ad8efbf07 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ef7ffb878cff42f2ac0d403ad8efbf07" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef7ffb878cff42f2ac0d403ad8efbf07" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 33947 } } }
I20260812 02:36:16.673789 3603 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef7ffb878cff42f2ac0d403ad8efbf07 [sys.catalog]: This master's current role is: LEADER
I20260812 02:36:16.673705 3612 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 02:36:16.677515 3612 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 02:36:16.683487 3612 catalog_manager.cc:1347] Generated new cluster ID: f70d24e214864a31976843d255da1366
I20260812 02:36:16.683672 3612 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 02:36:16.698199 3612 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 02:36:16.699297 3612 catalog_manager.cc:1504] Loading token signing keys...
I20260812 02:36:16.708248 3612 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ef7ffb878cff42f2ac0d403ad8efbf07: Generated new TSK 0
I20260812 02:36:16.709061 3612 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 02:36:16.712589 14748 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbep4M4/build/debug/bin/kudu
/tmp/dist-test-taskbep4M4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786501898713699-14748-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786501898713699-14748-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786501898713699-14748-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786501898713699-14748-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.103.1:0
--local_ip_for_outbound_sockets=127.14.103.1
--webserver_interface=127.14.103.1
--webserver_port=0
--tserver_master_addrs=127.14.103.62:33947
--builtin_ntp_servers=127.14.103.20:42751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260812 02:36:16.838951 3622 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:36:16.839277 3622 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:36:16.839320 3622 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:36:16.843674 3622 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:36:16.843746 3622 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:36:16.843806 3622 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.103.1
I20260812 02:36:16.848831 3622 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.103.20:42751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786501898713699-14748-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786501898713699-14748-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.103.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786501898713699-14748-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.103.1
--webserver_port=0
--tserver_master_addrs=127.14.103.62:33947
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3622
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.103.1
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786501898713699-14748-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:36:16.849973 3622 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:36:16.851543 3622 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260812 02:36:16.861848 3622 server_base.cc:1034] running on GCE node
W20260812 02:36:16.861848 3628 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:36:16.861884 3629 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:36:16.862094 3632 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:36:16.862569 3622 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:36:16.863313 3622 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:36:16.864497 3622 hybrid_clock.cc:648] HybridClock initialized: now 1786502176864468 us; error 46 us; skew 500 ppm
I20260812 02:36:16.866415 3622 webserver.cc:460] Webserver started at http://127.14.103.1:46165/ using document root <none> and password file <none>
I20260812 02:36:16.867084 3622 fs_manager.cc:359] Metadata directory not provided
I20260812 02:36:16.867185 3622 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:36:16.867431 3622 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:36:16.869705 3622 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786501898713699-14748-0/minicluster-data/ts-0/data/instance:
uuid: "6443362bb8e54a9f92661eb1b956df5e"
format_stamp: "Formatted at 2026-08-12 02:36:16 on dist-test-slave-2w3w"
server_key: "9c190f0b667b4bfff22f3700c0c4de2a"
server_key_iv: "6093254e591d7564c24896f6ba105072"
server_key_version: "encryptionkey@0"
I20260812 02:36:16.870308 3622 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786501898713699-14748-0/minicluster-data/ts-0/wal/instance:
uuid: "6443362bb8e54a9f92661eb1b956df5e"
format_stamp: "Formatted at 2026-08-12 02:36:16 on dist-test-slave-2w3w"
server_key: "9c190f0b667b4bfff22f3700c0c4de2a"
server_key_iv: "6093254e591d7564c24896f6ba105072"
server_key_version: "encryptionkey@0"
I20260812 02:36:16.874615 3622 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260812 02:36:16.877365 3638 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:36:16.878827 3622 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260812 02:36:16.879055 3622 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786501898713699-14748-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786501898713699-14748-0/minicluster-data/ts-0/wal
uuid: "6443362bb8e54a9f92661eb1b956df5e"
format_stamp: "Formatted at 2026-08-12 02:36:16 on dist-test-slave-2w3w"
server_key: "9c190f0b667b4bfff22f3700c0c4de2a"
server_key_iv: "6093254e591d7564c24896f6ba105072"
server_key_version: "encryptionkey@0"
I20260812 02:36:16.879199 3622 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786501898713699-14748-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786501898713699-14748-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786501898713699-14748-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:36:16.908046 3622 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:36:16.908752 3622 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:36:16.908928 3622 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:36:16.909511 3622 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:36:16.910457 3622 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:36:16.910507 3622 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:36:16.910547 3622 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:36:16.910562 3622 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:36:16.919816 3622 rpc_server.cc:307] RPC server started. Bound to: 127.14.103.1:42041
I20260812 02:36:16.919857 3773 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.103.1:42041 every 8 connection(s)
I20260812 02:36:16.920804 3622 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786501898713699-14748-0/minicluster-data/ts-0/data/info.pb
I20260812 02:36:16.921723 14748 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbep4M4/build/debug/bin/kudu as pid 3622
I20260812 02:36:16.921852 14748 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786501898713699-14748-0/minicluster-data/ts-0/wal/instance
I20260812 02:36:16.922227 14748 external_mini_cluster.cc:1428] Setting key b63325214c5161d5d8051d2aeaeef400
I20260812 02:36:16.924301 14748 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbep4M4/build/debug/bin/kudu
/tmp/dist-test-taskbep4M4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786501898713699-14748-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786501898713699-14748-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786501898713699-14748-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786501898713699-14748-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.103.2:0
--local_ip_for_outbound_sockets=127.14.103.2
--webserver_interface=127.14.103.2
--webserver_port=0
--tserver_master_addrs=127.14.103.62:33947
--builtin_ntp_servers=127.14.103.20:42751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260812 02:36:16.931686 3600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:16.922813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.1:43183 (local address 127.14.103.62:33947)
0812 02:36:16.922942 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0812 02:36:16.922949 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 02:36:16.924258 (+ 1309us) server_negotiation.cc:407] Connection header received
0812 02:36:16.925706 (+ 1448us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:36:16.925712 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:36:16.925778 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:36:16.925880 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:36:16.927240 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:16.928052 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:16.929268 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:16.929725 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:16.929866 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:36:16.931040 (+ 1174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:36:16.931046 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:36:16.931053 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:36:16.931073 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:36:16.931102 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:36:16.931117 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:36:16.931214 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0812 02:36:16.931366 (+ 152us) server_negotiation.cc:299] Negotiation successful
0812 02:36:16.931462 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260812 02:36:16.932487 3778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:16.923311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:33947 (local address 127.14.103.1:43183)
0812 02:36:16.924085 (+ 774us) negotiation.cc:107] Waiting for socket to connect
0812 02:36:16.924140 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0812 02:36:16.925492 (+ 1352us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:36:16.926140 (+ 648us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:36:16.926152 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:36:16.926274 (+ 122us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:36:16.927043 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:36:16.927065 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:36:16.928204 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:36:16.928206 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:36:16.929112 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:36:16.929126 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:36:16.929897 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:36:16.929900 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:36:16.930053 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:36:16.930724 (+ 671us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 02:36:16.930726 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:36:16.930767 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 02:36:16.930837 (+ 70us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:36:16.931213 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:36:16.931220 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:36:16.931225 (+ 5us) client_negotiation.cc:769] Sending connection context
0812 02:36:16.931329 (+ 104us) client_negotiation.cc:240] Negotiation successful
0812 02:36:16.931571 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":166,"threads_started":1}
I20260812 02:36:16.933920 3776 heartbeater.cc:347] Connected to a master server at 127.14.103.62:33947
I20260812 02:36:16.934186 3776 heartbeater.cc:464] Registering TS with master...
I20260812 02:36:16.934762 3776 heartbeater.cc:511] Master 127.14.103.62:33947 requested a full tablet report, sending...
I20260812 02:36:16.936488 3551 ts_manager.cc:194] Registered new tserver with Master: 6443362bb8e54a9f92661eb1b956df5e (127.14.103.1:42041)
I20260812 02:36:16.937721 3551 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.103.1:43183
W20260812 02:36:17.047525 3779 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:36:17.047829 3779 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:36:17.047919 3779 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:36:17.053409 3779 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:36:17.053500 3779 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:36:17.053594 3779 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.103.2
I20260812 02:36:17.059638 3779 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.103.20:42751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786501898713699-14748-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786501898713699-14748-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.103.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786501898713699-14748-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.103.2
--webserver_port=0
--tserver_master_addrs=127.14.103.62:33947
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3779
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.103.2
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786501898713699-14748-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:36:17.060875 3779 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:36:17.062371 3779 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:36:17.072719 3787 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:36:17.072719 3789 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:36:17.072774 3786 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:36:17.072961 3779 server_base.cc:1034] running on GCE node
I20260812 02:36:17.073498 3779 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:36:17.074302 3779 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:36:17.075522 3779 hybrid_clock.cc:648] HybridClock initialized: now 1786502177075487 us; error 52 us; skew 500 ppm
I20260812 02:36:17.077549 3779 webserver.cc:460] Webserver started at http://127.14.103.2:35183/ using document root <none> and password file <none>
I20260812 02:36:17.078182 3779 fs_manager.cc:359] Metadata directory not provided
I20260812 02:36:17.078287 3779 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:36:17.078531 3779 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:36:17.080708 3779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786501898713699-14748-0/minicluster-data/ts-1/data/instance:
uuid: "cdd90597ed584152abb9301e00df4b77"
format_stamp: "Formatted at 2026-08-12 02:36:17 on dist-test-slave-2w3w"
server_key: "7d000331891a8a581da7fd72d0a1fead"
server_key_iv: "dc04f5dfc626dc21e47c64d7cb8a9179"
server_key_version: "encryptionkey@0"
I20260812 02:36:17.081350 3779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786501898713699-14748-0/minicluster-data/ts-1/wal/instance:
uuid: "cdd90597ed584152abb9301e00df4b77"
format_stamp: "Formatted at 2026-08-12 02:36:17 on dist-test-slave-2w3w"
server_key: "7d000331891a8a581da7fd72d0a1fead"
server_key_iv: "dc04f5dfc626dc21e47c64d7cb8a9179"
server_key_version: "encryptionkey@0"
I20260812 02:36:17.085446 3779 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260812 02:36:17.088088 3795 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:36:17.089370 3779 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260812 02:36:17.089520 3779 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786501898713699-14748-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786501898713699-14748-0/minicluster-data/ts-1/wal
uuid: "cdd90597ed584152abb9301e00df4b77"
format_stamp: "Formatted at 2026-08-12 02:36:17 on dist-test-slave-2w3w"
server_key: "7d000331891a8a581da7fd72d0a1fead"
server_key_iv: "dc04f5dfc626dc21e47c64d7cb8a9179"
server_key_version: "encryptionkey@0"
I20260812 02:36:17.089648 3779 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786501898713699-14748-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786501898713699-14748-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786501898713699-14748-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:36:17.099766 3779 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:36:17.100437 3779 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:36:17.100643 3779 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:36:17.101249 3779 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:36:17.102207 3779 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:36:17.102280 3779 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:36:17.102357 3779 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:36:17.102407 3779 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:36:17.111608 3779 rpc_server.cc:307] RPC server started. Bound to: 127.14.103.2:38647
I20260812 02:36:17.111637 3925 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.103.2:38647 every 8 connection(s)
I20260812 02:36:17.112547 3779 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786501898713699-14748-0/minicluster-data/ts-1/data/info.pb
I20260812 02:36:17.112903 14748 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbep4M4/build/debug/bin/kudu as pid 3779
I20260812 02:36:17.113052 14748 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786501898713699-14748-0/minicluster-data/ts-1/wal/instance
I20260812 02:36:17.113445 14748 external_mini_cluster.cc:1428] Setting key 572a291ba330a072378dd758fa8bd487
I20260812 02:36:17.115890 14748 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbep4M4/build/debug/bin/kudu
/tmp/dist-test-taskbep4M4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786501898713699-14748-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786501898713699-14748-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786501898713699-14748-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786501898713699-14748-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.103.3:0
--local_ip_for_outbound_sockets=127.14.103.3
--webserver_interface=127.14.103.3
--webserver_port=0
--tserver_master_addrs=127.14.103.62:33947
--builtin_ntp_servers=127.14.103.20:42751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260812 02:36:17.122941 3600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:17.114596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.2:39799 (local address 127.14.103.62:33947)
0812 02:36:17.114755 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0812 02:36:17.114760 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:36:17.116163 (+ 1403us) server_negotiation.cc:407] Connection header received
0812 02:36:17.117670 (+ 1507us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:36:17.117674 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:36:17.117736 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:36:17.117821 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:36:17.118784 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:17.119514 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:17.120792 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:17.121040 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:17.121159 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:36:17.122390 (+ 1231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:36:17.122394 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:36:17.122396 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:36:17.122414 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:36:17.122440 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:36:17.122449 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:36:17.122534 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0812 02:36:17.122706 (+ 172us) server_negotiation.cc:299] Negotiation successful
0812 02:36:17.122777 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260812 02:36:17.123966 3928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:17.115252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:33947 (local address 127.14.103.2:39799)
0812 02:36:17.115991 (+ 739us) negotiation.cc:107] Waiting for socket to connect
0812 02:36:17.116055 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0812 02:36:17.117449 (+ 1394us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:36:17.117995 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:36:17.118002 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:36:17.118089 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:36:17.118599 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:36:17.118611 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:36:17.119677 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:36:17.119680 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:36:17.120651 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:36:17.120661 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:36:17.121207 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:36:17.121211 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:36:17.121390 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:36:17.122100 (+ 710us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 02:36:17.122102 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:36:17.122146 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 02:36:17.122219 (+ 73us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:36:17.122535 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:36:17.122548 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:36:17.122557 (+ 9us) client_negotiation.cc:769] Sending connection context
0812 02:36:17.122703 (+ 146us) client_negotiation.cc:240] Negotiation successful
0812 02:36:17.123041 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":150,"threads_started":1}
I20260812 02:36:17.125309 3926 heartbeater.cc:347] Connected to a master server at 127.14.103.62:33947
I20260812 02:36:17.125591 3926 heartbeater.cc:464] Registering TS with master...
I20260812 02:36:17.126144 3926 heartbeater.cc:511] Master 127.14.103.62:33947 requested a full tablet report, sending...
I20260812 02:36:17.127300 3551 ts_manager.cc:194] Registered new tserver with Master: cdd90597ed584152abb9301e00df4b77 (127.14.103.2:38647)
I20260812 02:36:17.127928 3551 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.103.2:39799
W20260812 02:36:17.233611 3929 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:36:17.233912 3929 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:36:17.233953 3929 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:36:17.238210 3929 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:36:17.238297 3929 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:36:17.238355 3929 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.103.3
I20260812 02:36:17.243453 3929 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.103.20:42751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786501898713699-14748-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786501898713699-14748-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.103.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786501898713699-14748-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.103.3
--webserver_port=0
--tserver_master_addrs=127.14.103.62:33947
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3929
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.103.3
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786501898713699-14748-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:36:17.244702 3929 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:36:17.246206 3929 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:36:17.256143 3934 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:36:17.256139 3937 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:36:17.256140 3935 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:36:17.256378 3929 server_base.cc:1034] running on GCE node
I20260812 02:36:17.256875 3929 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:36:17.257580 3929 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:36:17.258773 3929 hybrid_clock.cc:648] HybridClock initialized: now 1786502177258739 us; error 58 us; skew 500 ppm
I20260812 02:36:17.260695 3929 webserver.cc:460] Webserver started at http://127.14.103.3:35875/ using document root <none> and password file <none>
I20260812 02:36:17.261281 3929 fs_manager.cc:359] Metadata directory not provided
I20260812 02:36:17.261374 3929 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:36:17.261621 3929 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:36:17.263763 3929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786501898713699-14748-0/minicluster-data/ts-2/data/instance:
uuid: "c083f474376a432cb7966a18f399e5df"
format_stamp: "Formatted at 2026-08-12 02:36:17 on dist-test-slave-2w3w"
server_key: "dbb5961374e381caf14e1f630924c7cc"
server_key_iv: "b04982b7b2491ae7a84023891e9101af"
server_key_version: "encryptionkey@0"
I20260812 02:36:17.264325 3929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786501898713699-14748-0/minicluster-data/ts-2/wal/instance:
uuid: "c083f474376a432cb7966a18f399e5df"
format_stamp: "Formatted at 2026-08-12 02:36:17 on dist-test-slave-2w3w"
server_key: "dbb5961374e381caf14e1f630924c7cc"
server_key_iv: "b04982b7b2491ae7a84023891e9101af"
server_key_version: "encryptionkey@0"
I20260812 02:36:17.268044 3929 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260812 02:36:17.270589 3944 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:36:17.271840 3929 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260812 02:36:17.271973 3929 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786501898713699-14748-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786501898713699-14748-0/minicluster-data/ts-2/wal
uuid: "c083f474376a432cb7966a18f399e5df"
format_stamp: "Formatted at 2026-08-12 02:36:17 on dist-test-slave-2w3w"
server_key: "dbb5961374e381caf14e1f630924c7cc"
server_key_iv: "b04982b7b2491ae7a84023891e9101af"
server_key_version: "encryptionkey@0"
I20260812 02:36:17.272107 3929 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786501898713699-14748-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786501898713699-14748-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786501898713699-14748-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:36:17.289705 3929 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:36:17.290503 3929 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:36:17.290729 3929 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:36:17.291399 3929 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:36:17.292443 3929 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:36:17.292526 3929 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:36:17.292601 3929 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:36:17.292646 3929 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:36:17.302490 3929 rpc_server.cc:307] RPC server started. Bound to: 127.14.103.3:44323
I20260812 02:36:17.302520 4078 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.103.3:44323 every 8 connection(s)
I20260812 02:36:17.303434 3929 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786501898713699-14748-0/minicluster-data/ts-2/data/info.pb
I20260812 02:36:17.304405 14748 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbep4M4/build/debug/bin/kudu as pid 3929
I20260812 02:36:17.304546 14748 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786501898713699-14748-0/minicluster-data/ts-2/wal/instance
I20260812 02:36:17.304867 14748 external_mini_cluster.cc:1428] Setting key f19fbc395ec9abe0db643549230eede6
I20260812 02:36:17.313804 3600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:17.305645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.3:60785 (local address 127.14.103.62:33947)
0812 02:36:17.307364 (+ 1719us) server_negotiation.cc:206] Beginning negotiation
0812 02:36:17.307369 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:36:17.307396 (+ 27us) server_negotiation.cc:407] Connection header received
0812 02:36:17.308696 (+ 1300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:36:17.308699 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:36:17.308747 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:36:17.308830 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:36:17.309749 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:17.310480 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:17.311783 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:17.312079 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:17.312177 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:36:17.313347 (+ 1170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:36:17.313350 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:36:17.313352 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:36:17.313366 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:36:17.313388 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:36:17.313397 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:36:17.313465 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0812 02:36:17.313610 (+ 145us) server_negotiation.cc:299] Negotiation successful
0812 02:36:17.313671 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1621}
I20260812 02:36:17.314782 4081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:17.306229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:33947 (local address 127.14.103.3:60785)
0812 02:36:17.307041 (+ 812us) negotiation.cc:107] Waiting for socket to connect
0812 02:36:17.307103 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0812 02:36:17.308480 (+ 1377us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:36:17.308993 (+ 513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:36:17.309000 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:36:17.309083 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:36:17.309596 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:36:17.309609 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:36:17.310619 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:36:17.310622 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:36:17.311645 (+ 1023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:36:17.311656 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:36:17.312254 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:36:17.312258 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:36:17.312434 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:36:17.313068 (+ 634us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 02:36:17.313069 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:36:17.313113 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 02:36:17.313183 (+ 70us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:36:17.313479 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:36:17.313486 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:36:17.313490 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 02:36:17.313590 (+ 100us) client_negotiation.cc:240] Negotiation successful
0812 02:36:17.313845 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":179,"threads_started":1}
I20260812 02:36:17.316130 4079 heartbeater.cc:347] Connected to a master server at 127.14.103.62:33947
I20260812 02:36:17.316422 4079 heartbeater.cc:464] Registering TS with master...
I20260812 02:36:17.317003 4079 heartbeater.cc:511] Master 127.14.103.62:33947 requested a full tablet report, sending...
I20260812 02:36:17.318328 3551 ts_manager.cc:194] Registered new tserver with Master: c083f474376a432cb7966a18f399e5df (127.14.103.3:44323)
I20260812 02:36:17.319195 3551 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.103.3:60785
I20260812 02:36:17.319523 14748 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 02:36:17.325970 3600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:17.321744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52270 (local address 127.14.103.62:33947)
0812 02:36:17.321923 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0812 02:36:17.321927 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:36:17.322091 (+ 164us) server_negotiation.cc:407] Connection header received
0812 02:36:17.322162 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:36:17.322164 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:36:17.322204 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:36:17.322274 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:36:17.322538 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:17.323141 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:17.324054 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:17.324274 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:17.324403 (+ 129us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:36:17.325360 (+ 957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:36:17.325364 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:36:17.325366 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:36:17.325378 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:36:17.325397 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:36:17.325405 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:36:17.325525 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0812 02:36:17.325706 (+ 181us) server_negotiation.cc:299] Negotiation successful
0812 02:36:17.325787 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260812 02:36:17.329118 3551 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52270:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 02:36:17.332473 3551 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 02:36:17.346997 4090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:17.340852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.2:38647 (local address 127.0.0.1:46806)
0812 02:36:17.341177 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0812 02:36:17.341227 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0812 02:36:17.341360 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:36:17.342532 (+ 1172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:36:17.342539 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:36:17.342563 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:36:17.342721 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:36:17.342728 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:36:17.344036 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:36:17.344043 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:36:17.345272 (+ 1229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:36:17.345282 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:36:17.346362 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:36:17.346366 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:36:17.346498 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:36:17.346540 (+ 42us) client_negotiation.cc:769] Sending connection context
0812 02:36:17.346651 (+ 111us) client_negotiation.cc:240] Negotiation successful
0812 02:36:17.346783 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"mutex_wait_us":77,"thread_start_us":78,"threads_started":1}
I20260812 02:36:17.347036 4094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:17.341536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46806 (local address 127.14.103.2:38647)
0812 02:36:17.341977 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0812 02:36:17.341990 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0812 02:36:17.342016 (+ 26us) server_negotiation.cc:407] Connection header received
0812 02:36:17.342100 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:36:17.342106 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:36:17.342328 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:36:17.342889 (+ 561us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:36:17.343069 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:17.343913 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:17.345447 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:17.346221 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:17.346346 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:36:17.346576 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0812 02:36:17.346699 (+ 123us) server_negotiation.cc:299] Negotiation successful
0812 02:36:17.346819 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":108,"threads_started":1}
I20260812 02:36:17.347745 4095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:17.341684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47838 (local address 127.14.103.3:44323)
0812 02:36:17.342136 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0812 02:36:17.342150 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0812 02:36:17.342528 (+ 378us) server_negotiation.cc:407] Connection header received
0812 02:36:17.342621 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:36:17.342626 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:36:17.342863 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:36:17.343041 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:36:17.343317 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:17.344566 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:17.346169 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:17.346940 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:17.347248 (+ 308us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:36:17.347442 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0812 02:36:17.347491 (+ 49us) server_negotiation.cc:299] Negotiation successful
0812 02:36:17.347584 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":119,"threads_started":1}
I20260812 02:36:17.347733 4091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:17.340915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.1:42041 (local address 127.0.0.1:57896)
0812 02:36:17.341614 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0812 02:36:17.341632 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 02:36:17.341727 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:36:17.342887 (+ 1160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:36:17.342891 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:36:17.342910 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:36:17.343053 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:36:17.343061 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:36:17.344701 (+ 1640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:36:17.344704 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:36:17.346003 (+ 1299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:36:17.346014 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:36:17.347256 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:36:17.347261 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:36:17.347377 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:36:17.347404 (+ 27us) client_negotiation.cc:769] Sending connection context
0812 02:36:17.347456 (+ 52us) client_negotiation.cc:240] Negotiation successful
0812 02:36:17.347529 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":622,"mutex_wait_us":38,"thread_start_us":138,"threads_started":1}
I20260812 02:36:17.347724 4092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:17.341163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.3:44323 (local address 127.0.0.1:47838)
0812 02:36:17.342267 (+ 1104us) negotiation.cc:107] Waiting for socket to connect
0812 02:36:17.342282 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 02:36:17.342406 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:36:17.343076 (+ 670us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:36:17.343079 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:36:17.343097 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:36:17.343184 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:36:17.343189 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:36:17.344849 (+ 1660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:36:17.344852 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:36:17.346028 (+ 1176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:36:17.346035 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:36:17.347231 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:36:17.347235 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:36:17.347320 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:36:17.347341 (+ 21us) client_negotiation.cc:769] Sending connection context
0812 02:36:17.347406 (+ 65us) client_negotiation.cc:240] Negotiation successful
0812 02:36:17.347495 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1040,"mutex_wait_us":67,"thread_start_us":97,"threads_started":1}
I20260812 02:36:17.348536 4096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:17.341863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57896 (local address 127.14.103.1:42041)
0812 02:36:17.342442 (+ 579us) server_negotiation.cc:206] Beginning negotiation
0812 02:36:17.342456 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0812 02:36:17.342480 (+ 24us) server_negotiation.cc:407] Connection header received
0812 02:36:17.342555 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:36:17.342559 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:36:17.342726 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:36:17.342886 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:36:17.343203 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:17.344555 (+ 1352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:17.346157 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:17.347094 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:17.347818 (+ 724us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:36:17.348023 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0812 02:36:17.348076 (+ 53us) server_negotiation.cc:299] Negotiation successful
0812 02:36:17.348344 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":456,"thread_start_us":127,"threads_started":1}
I20260812 02:36:17.351052 4002 tablet_service.cc:1467] Processing CreateTablet for tablet a6f93e3bb66e44889e81de27fd0eaf0a (DEFAULT_TABLE table=test-table [id=99a800d1fc4c4fedad0c56a9588efb03]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:36:17.351050 3852 tablet_service.cc:1467] Processing CreateTablet for tablet a6f93e3bb66e44889e81de27fd0eaf0a (DEFAULT_TABLE table=test-table [id=99a800d1fc4c4fedad0c56a9588efb03]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:36:17.352325 3852 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6f93e3bb66e44889e81de27fd0eaf0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:36:17.352325 4002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6f93e3bb66e44889e81de27fd0eaf0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:36:17.353837 3699 tablet_service.cc:1467] Processing CreateTablet for tablet a6f93e3bb66e44889e81de27fd0eaf0a (DEFAULT_TABLE table=test-table [id=99a800d1fc4c4fedad0c56a9588efb03]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:36:17.354854 3699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6f93e3bb66e44889e81de27fd0eaf0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:36:17.361035 4097 tablet_bootstrap.cc:492] T a6f93e3bb66e44889e81de27fd0eaf0a P cdd90597ed584152abb9301e00df4b77: Bootstrap starting.
I20260812 02:36:17.361649 4098 tablet_bootstrap.cc:492] T a6f93e3bb66e44889e81de27fd0eaf0a P c083f474376a432cb7966a18f399e5df: Bootstrap starting.
I20260812 02:36:17.362493 4099 tablet_bootstrap.cc:492] T a6f93e3bb66e44889e81de27fd0eaf0a P 6443362bb8e54a9f92661eb1b956df5e: Bootstrap starting.
I20260812 02:36:17.364450 4098 tablet_bootstrap.cc:654] T a6f93e3bb66e44889e81de27fd0eaf0a P c083f474376a432cb7966a18f399e5df: Neither blocks nor log segments found. Creating new log.
I20260812 02:36:17.364450 4097 tablet_bootstrap.cc:654] T a6f93e3bb66e44889e81de27fd0eaf0a P cdd90597ed584152abb9301e00df4b77: Neither blocks nor log segments found. Creating new log.
I20260812 02:36:17.364753 4099 tablet_bootstrap.cc:654] T a6f93e3bb66e44889e81de27fd0eaf0a P 6443362bb8e54a9f92661eb1b956df5e: Neither blocks nor log segments found. Creating new log.
I20260812 02:36:17.365389 4098 log.cc:824] T a6f93e3bb66e44889e81de27fd0eaf0a P c083f474376a432cb7966a18f399e5df: Log is configured to *not* fsync() on all Append() calls
I20260812 02:36:17.365389 4097 log.cc:824] T a6f93e3bb66e44889e81de27fd0eaf0a P cdd90597ed584152abb9301e00df4b77: Log is configured to *not* fsync() on all Append() calls
I20260812 02:36:17.365513 4099 log.cc:824] T a6f93e3bb66e44889e81de27fd0eaf0a P 6443362bb8e54a9f92661eb1b956df5e: Log is configured to *not* fsync() on all Append() calls
I20260812 02:36:17.367534 4097 tablet_bootstrap.cc:492] T a6f93e3bb66e44889e81de27fd0eaf0a P cdd90597ed584152abb9301e00df4b77: No bootstrap required, opened a new log
I20260812 02:36:17.367631 4098 tablet_bootstrap.cc:492] T a6f93e3bb66e44889e81de27fd0eaf0a P c083f474376a432cb7966a18f399e5df: No bootstrap required, opened a new log
I20260812 02:36:17.367794 4097 ts_tablet_manager.cc:1397] T a6f93e3bb66e44889e81de27fd0eaf0a P cdd90597ed584152abb9301e00df4b77: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260812 02:36:17.367841 4098 ts_tablet_manager.cc:1397] T a6f93e3bb66e44889e81de27fd0eaf0a P c083f474376a432cb7966a18f399e5df: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260812 02:36:17.368526 4099 tablet_bootstrap.cc:492] T a6f93e3bb66e44889e81de27fd0eaf0a P 6443362bb8e54a9f92661eb1b956df5e: No bootstrap required, opened a new log
I20260812 02:36:17.368724 4099 ts_tablet_manager.cc:1397] T a6f93e3bb66e44889e81de27fd0eaf0a P 6443362bb8e54a9f92661eb1b956df5e: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260812 02:36:17.371348 4097 raft_consensus.cc:348] T a6f93e3bb66e44889e81de27fd0eaf0a P cdd90597ed584152abb9301e00df4b77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdd90597ed584152abb9301e00df4b77" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 38647 } } peers { permanent_uuid: "6443362bb8e54a9f92661eb1b956df5e" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 42041 } } peers { permanent_uuid: "c083f474376a432cb7966a18f399e5df" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 44323 } }
I20260812 02:36:17.371642 4097 raft_consensus.cc:374] T a6f93e3bb66e44889e81de27fd0eaf0a P cdd90597ed584152abb9301e00df4b77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:36:17.371706 4097 raft_consensus.cc:729] T a6f93e3bb66e44889e81de27fd0eaf0a P cdd90597ed584152abb9301e00df4b77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cdd90597ed584152abb9301e00df4b77, State: Initialized, Role: FOLLOWER
I20260812 02:36:17.372226 4097 consensus_queue.cc:260] T a6f93e3bb66e44889e81de27fd0eaf0a P cdd90597ed584152abb9301e00df4b77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdd90597ed584152abb9301e00df4b77" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 38647 } } peers { permanent_uuid: "6443362bb8e54a9f92661eb1b956df5e" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 42041 } } peers { permanent_uuid: "c083f474376a432cb7966a18f399e5df" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 44323 } }
I20260812 02:36:17.372962 3926 heartbeater.cc:503] Master 127.14.103.62:33947 was elected leader, sending a full tablet report...
I20260812 02:36:17.373312 4097 ts_tablet_manager.cc:1428] T a6f93e3bb66e44889e81de27fd0eaf0a P cdd90597ed584152abb9301e00df4b77: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260812 02:36:17.373950 4098 raft_consensus.cc:348] T a6f93e3bb66e44889e81de27fd0eaf0a P c083f474376a432cb7966a18f399e5df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdd90597ed584152abb9301e00df4b77" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 38647 } } peers { permanent_uuid: "6443362bb8e54a9f92661eb1b956df5e" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 42041 } } peers { permanent_uuid: "c083f474376a432cb7966a18f399e5df" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 44323 } }
I20260812 02:36:17.373950 4099 raft_consensus.cc:348] T a6f93e3bb66e44889e81de27fd0eaf0a P 6443362bb8e54a9f92661eb1b956df5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdd90597ed584152abb9301e00df4b77" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 38647 } } peers { permanent_uuid: "6443362bb8e54a9f92661eb1b956df5e" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 42041 } } peers { permanent_uuid: "c083f474376a432cb7966a18f399e5df" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 44323 } }
I20260812 02:36:17.374256 4099 raft_consensus.cc:374] T a6f93e3bb66e44889e81de27fd0eaf0a P 6443362bb8e54a9f92661eb1b956df5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:36:17.374239 4098 raft_consensus.cc:374] T a6f93e3bb66e44889e81de27fd0eaf0a P c083f474376a432cb7966a18f399e5df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:36:17.374326 4099 raft_consensus.cc:729] T a6f93e3bb66e44889e81de27fd0eaf0a P 6443362bb8e54a9f92661eb1b956df5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6443362bb8e54a9f92661eb1b956df5e, State: Initialized, Role: FOLLOWER
I20260812 02:36:17.374374 4098 raft_consensus.cc:729] T a6f93e3bb66e44889e81de27fd0eaf0a P c083f474376a432cb7966a18f399e5df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c083f474376a432cb7966a18f399e5df, State: Initialized, Role: FOLLOWER
I20260812 02:36:17.374855 4099 consensus_queue.cc:260] T a6f93e3bb66e44889e81de27fd0eaf0a P 6443362bb8e54a9f92661eb1b956df5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdd90597ed584152abb9301e00df4b77" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 38647 } } peers { permanent_uuid: "6443362bb8e54a9f92661eb1b956df5e" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 42041 } } peers { permanent_uuid: "c083f474376a432cb7966a18f399e5df" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 44323 } }
I20260812 02:36:17.374855 4098 consensus_queue.cc:260] T a6f93e3bb66e44889e81de27fd0eaf0a P c083f474376a432cb7966a18f399e5df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdd90597ed584152abb9301e00df4b77" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 38647 } } peers { permanent_uuid: "6443362bb8e54a9f92661eb1b956df5e" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 42041 } } peers { permanent_uuid: "c083f474376a432cb7966a18f399e5df" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 44323 } }
I20260812 02:36:17.375870 3776 heartbeater.cc:503] Master 127.14.103.62:33947 was elected leader, sending a full tablet report...
I20260812 02:36:17.376039 4099 ts_tablet_manager.cc:1428] T a6f93e3bb66e44889e81de27fd0eaf0a P 6443362bb8e54a9f92661eb1b956df5e: Time spent starting tablet: real 0.007s user 0.009s sys 0.000s
I20260812 02:36:17.376039 4098 ts_tablet_manager.cc:1428] T a6f93e3bb66e44889e81de27fd0eaf0a P c083f474376a432cb7966a18f399e5df: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20260812 02:36:17.376981 4079 heartbeater.cc:503] Master 127.14.103.62:33947 was elected leader, sending a full tablet report...
W20260812 02:36:17.422075 3777 tablet.cc:2369] T a6f93e3bb66e44889e81de27fd0eaf0a P 6443362bb8e54a9f92661eb1b956df5e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 02:36:17.436770 4106 raft_consensus.cc:482] T a6f93e3bb66e44889e81de27fd0eaf0a P c083f474376a432cb7966a18f399e5df [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 02:36:17.437038 4106 raft_consensus.cc:504] T a6f93e3bb66e44889e81de27fd0eaf0a P c083f474376a432cb7966a18f399e5df [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdd90597ed584152abb9301e00df4b77" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 38647 } } peers { permanent_uuid: "6443362bb8e54a9f92661eb1b956df5e" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 42041 } } peers { permanent_uuid: "c083f474376a432cb7966a18f399e5df" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 44323 } }
I20260812 02:36:17.438387 4106 leader_election.cc:288] T a6f93e3bb66e44889e81de27fd0eaf0a P c083f474376a432cb7966a18f399e5df [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cdd90597ed584152abb9301e00df4b77 (127.14.103.2:38647), 6443362bb8e54a9f92661eb1b956df5e (127.14.103.1:42041)
I20260812 02:36:17.442847 4081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:17.438610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.2:38647 (local address 127.14.103.3:42559)
0812 02:36:17.438772 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0812 02:36:17.438789 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 02:36:17.438911 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:36:17.439286 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:36:17.439289 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:36:17.439315 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:36:17.439455 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:36:17.439465 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:36:17.440492 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:36:17.440496 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:36:17.441395 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:36:17.441404 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:36:17.442349 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:36:17.442353 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:36:17.442468 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:36:17.442496 (+ 28us) client_negotiation.cc:769] Sending connection context
0812 02:36:17.442599 (+ 103us) client_negotiation.cc:240] Negotiation successful
0812 02:36:17.442687 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62,"mutex_wait_us":23}
I20260812 02:36:17.442994 4094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:17.438743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.3:42559 (local address 127.14.103.2:38647)
0812 02:36:17.438915 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0812 02:36:17.438920 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:36:17.438941 (+ 21us) server_negotiation.cc:407] Connection header received
0812 02:36:17.439104 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:36:17.439111 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:36:17.439178 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:36:17.439274 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:36:17.439571 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:17.440345 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:17.441527 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:17.442178 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:17.442306 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:36:17.442356 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0812 02:36:17.442671 (+ 315us) server_negotiation.cc:299] Negotiation successful
0812 02:36:17.442813 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260812 02:36:17.443171 4107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:17.438734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.1:42041 (local address 127.14.103.3:42179)
0812 02:36:17.439207 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0812 02:36:17.439223 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 02:36:17.439405 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:36:17.439738 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:36:17.439741 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:36:17.439761 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:36:17.439898 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:36:17.439909 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:36:17.440852 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:36:17.440855 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:36:17.441827 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:36:17.441838 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:36:17.442695 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:36:17.442699 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:36:17.442801 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:36:17.442826 (+ 25us) client_negotiation.cc:769] Sending connection context
0812 02:36:17.442910 (+ 84us) client_negotiation.cc:240] Negotiation successful
0812 02:36:17.443004 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":387,"mutex_wait_us":22,"thread_start_us":119,"threads_started":1}
I20260812 02:36:17.443218 4096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:17.438923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.3:42179 (local address 127.14.103.1:42041)
0812 02:36:17.439092 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0812 02:36:17.439097 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:36:17.439311 (+ 214us) server_negotiation.cc:407] Connection header received
0812 02:36:17.439520 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:36:17.439524 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:36:17.439585 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:36:17.439658 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:36:17.440039 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:17.440717 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:17.441995 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:17.442529 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:17.442650 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:36:17.442694 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0812 02:36:17.442982 (+ 288us) server_negotiation.cc:299] Negotiation successful
0812 02:36:17.443073 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260812 02:36:17.443818 3874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6f93e3bb66e44889e81de27fd0eaf0a" candidate_uuid: "c083f474376a432cb7966a18f399e5df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdd90597ed584152abb9301e00df4b77" is_pre_election: true
I20260812 02:36:17.443811 3721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6f93e3bb66e44889e81de27fd0eaf0a" candidate_uuid: "c083f474376a432cb7966a18f399e5df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6443362bb8e54a9f92661eb1b956df5e" is_pre_election: true
I20260812 02:36:17.444142 3874 raft_consensus.cc:2454] T a6f93e3bb66e44889e81de27fd0eaf0a P cdd90597ed584152abb9301e00df4b77 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c083f474376a432cb7966a18f399e5df in term 0.
I20260812 02:36:17.444149 3721 raft_consensus.cc:2454] T a6f93e3bb66e44889e81de27fd0eaf0a P 6443362bb8e54a9f92661eb1b956df5e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c083f474376a432cb7966a18f399e5df in term 0.
I20260812 02:36:17.444679 3947 leader_election.cc:302] T a6f93e3bb66e44889e81de27fd0eaf0a P c083f474376a432cb7966a18f399e5df [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c083f474376a432cb7966a18f399e5df, cdd90597ed584152abb9301e00df4b77; no voters:
I20260812 02:36:17.445016 4106 raft_consensus.cc:2781] T a6f93e3bb66e44889e81de27fd0eaf0a P c083f474376a432cb7966a18f399e5df [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 02:36:17.445106 4106 raft_consensus.cc:482] T a6f93e3bb66e44889e81de27fd0eaf0a P c083f474376a432cb7966a18f399e5df [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 02:36:17.445137 4106 raft_consensus.cc:3037] T a6f93e3bb66e44889e81de27fd0eaf0a P c083f474376a432cb7966a18f399e5df [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:36:17.446259 4106 raft_consensus.cc:504] T a6f93e3bb66e44889e81de27fd0eaf0a P c083f474376a432cb7966a18f399e5df [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdd90597ed584152abb9301e00df4b77" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 38647 } } peers { permanent_uuid: "6443362bb8e54a9f92661eb1b956df5e" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 42041 } } peers { permanent_uuid: "c083f474376a432cb7966a18f399e5df" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 44323 } }
I20260812 02:36:17.446669 4106 leader_election.cc:288] T a6f93e3bb66e44889e81de27fd0eaf0a P c083f474376a432cb7966a18f399e5df [CANDIDATE]: Term 1 election: Requested vote from peers cdd90597ed584152abb9301e00df4b77 (127.14.103.2:38647), 6443362bb8e54a9f92661eb1b956df5e (127.14.103.1:42041)
I20260812 02:36:17.447100 3721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6f93e3bb66e44889e81de27fd0eaf0a" candidate_uuid: "c083f474376a432cb7966a18f399e5df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6443362bb8e54a9f92661eb1b956df5e"
I20260812 02:36:17.447136 3874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6f93e3bb66e44889e81de27fd0eaf0a" candidate_uuid: "c083f474376a432cb7966a18f399e5df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdd90597ed584152abb9301e00df4b77"
I20260812 02:36:17.447273 3721 raft_consensus.cc:3037] T a6f93e3bb66e44889e81de27fd0eaf0a P 6443362bb8e54a9f92661eb1b956df5e [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:36:17.447312 3874 raft_consensus.cc:3037] T a6f93e3bb66e44889e81de27fd0eaf0a P cdd90597ed584152abb9301e00df4b77 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:36:17.448410 3874 raft_consensus.cc:2454] T a6f93e3bb66e44889e81de27fd0eaf0a P cdd90597ed584152abb9301e00df4b77 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c083f474376a432cb7966a18f399e5df in term 1.
I20260812 02:36:17.448414 3721 raft_consensus.cc:2454] T a6f93e3bb66e44889e81de27fd0eaf0a P 6443362bb8e54a9f92661eb1b956df5e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c083f474376a432cb7966a18f399e5df in term 1.
I20260812 02:36:17.448777 3948 leader_election.cc:302] T a6f93e3bb66e44889e81de27fd0eaf0a P c083f474376a432cb7966a18f399e5df [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6443362bb8e54a9f92661eb1b956df5e, c083f474376a432cb7966a18f399e5df; no voters:
I20260812 02:36:17.448989 4106 raft_consensus.cc:2781] T a6f93e3bb66e44889e81de27fd0eaf0a P c083f474376a432cb7966a18f399e5df [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:36:17.449254 4106 raft_consensus.cc:686] T a6f93e3bb66e44889e81de27fd0eaf0a P c083f474376a432cb7966a18f399e5df [term 1 LEADER]: Becoming Leader. State: Replica: c083f474376a432cb7966a18f399e5df, State: Running, Role: LEADER
I20260812 02:36:17.449606 4106 consensus_queue.cc:237] T a6f93e3bb66e44889e81de27fd0eaf0a P c083f474376a432cb7966a18f399e5df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdd90597ed584152abb9301e00df4b77" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 38647 } } peers { permanent_uuid: "6443362bb8e54a9f92661eb1b956df5e" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 42041 } } peers { permanent_uuid: "c083f474376a432cb7966a18f399e5df" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 44323 } }
I20260812 02:36:17.452778 3550 catalog_manager.cc:5495] T a6f93e3bb66e44889e81de27fd0eaf0a P c083f474376a432cb7966a18f399e5df reported cstate change: term changed from 0 to 1, leader changed from <none> to c083f474376a432cb7966a18f399e5df (127.14.103.3). New cstate: current_term: 1 leader_uuid: "c083f474376a432cb7966a18f399e5df" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdd90597ed584152abb9301e00df4b77" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 38647 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6443362bb8e54a9f92661eb1b956df5e" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 42041 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c083f474376a432cb7966a18f399e5df" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 44323 } health_report { overall_health: HEALTHY } } }
I20260812 02:36:17.473315 4095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:17.469564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47854 (local address 127.14.103.3:44323)
0812 02:36:17.469689 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0812 02:36:17.469692 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 02:36:17.469708 (+ 16us) server_negotiation.cc:407] Connection header received
0812 02:36:17.469776 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:36:17.469778 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:36:17.469831 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:36:17.469918 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 02:36:17.470146 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:17.470641 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:17.471648 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:17.471858 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:17.471939 (+ 81us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:36:17.472345 (+ 406us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 02:36:17.472851 (+ 506us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 02:36:17.473000 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0812 02:36:17.473101 (+ 101us) server_negotiation.cc:299] Negotiation successful
0812 02:36:17.473143 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260812 02:36:17.479894 3721 raft_consensus.cc:1261] T a6f93e3bb66e44889e81de27fd0eaf0a P 6443362bb8e54a9f92661eb1b956df5e [term 1 FOLLOWER]: Refusing update from remote peer c083f474376a432cb7966a18f399e5df: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:36:17.479894 3874 raft_consensus.cc:1261] T a6f93e3bb66e44889e81de27fd0eaf0a P cdd90597ed584152abb9301e00df4b77 [term 1 FOLLOWER]: Refusing update from remote peer c083f474376a432cb7966a18f399e5df: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:36:17.480722 4109 consensus_queue.cc:1035] T a6f93e3bb66e44889e81de27fd0eaf0a P c083f474376a432cb7966a18f399e5df [LEADER]: Connected to new peer: Peer: permanent_uuid: "6443362bb8e54a9f92661eb1b956df5e" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 42041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:36:17.480978 4106 consensus_queue.cc:1035] T a6f93e3bb66e44889e81de27fd0eaf0a P c083f474376a432cb7966a18f399e5df [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdd90597ed584152abb9301e00df4b77" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 38647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:36:17.488629 4112 mvcc.cc:205] Tried to move back new op lower bound from 7317512918949216256 to 7317512918836195328. Current Snapshot: MvccSnapshot[applied={T|T < 7317512918949216256}]
I20260812 02:36:17.488843 4115 mvcc.cc:205] Tried to move back new op lower bound from 7317512918949216256 to 7317512918836195328. Current Snapshot: MvccSnapshot[applied={T|T < 7317512918949216256}]
I20260812 02:36:17.491552 4114 mvcc.cc:205] Tried to move back new op lower bound from 7317512918949216256 to 7317512918836195328. Current Snapshot: MvccSnapshot[applied={T|T < 7317512918949216256}]
I20260812 02:36:17.497161 3551 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52270:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 02:36:17.497336 3551 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52270:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 02:36:17.499866 3551 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ef7ffb878cff42f2ac0d403ad8efbf07: Sending DeleteTablet for 3 replicas of tablet a6f93e3bb66e44889e81de27fd0eaf0a
I20260812 02:36:17.500692 3852 tablet_service.cc:1514] Processing DeleteTablet for tablet a6f93e3bb66e44889e81de27fd0eaf0a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:36:17 UTC) from {username='slave'} at 127.0.0.1:46806
I20260812 02:36:17.500692 4002 tablet_service.cc:1514] Processing DeleteTablet for tablet a6f93e3bb66e44889e81de27fd0eaf0a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:36:17 UTC) from {username='slave'} at 127.0.0.1:47838
I20260812 02:36:17.500741 3699 tablet_service.cc:1514] Processing DeleteTablet for tablet a6f93e3bb66e44889e81de27fd0eaf0a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:36:17 UTC) from {username='slave'} at 127.0.0.1:57896
I20260812 02:36:17.501255 4127 tablet_replica.cc:330] T a6f93e3bb66e44889e81de27fd0eaf0a P c083f474376a432cb7966a18f399e5df: stopping tablet replica
I20260812 02:36:17.501437 4129 tablet_replica.cc:330] T a6f93e3bb66e44889e81de27fd0eaf0a P 6443362bb8e54a9f92661eb1b956df5e: stopping tablet replica
I20260812 02:36:17.501705 4127 raft_consensus.cc:2229] T a6f93e3bb66e44889e81de27fd0eaf0a P c083f474376a432cb7966a18f399e5df [term 1 LEADER]: Raft consensus shutting down.
I20260812 02:36:17.501744 4129 raft_consensus.cc:2229] T a6f93e3bb66e44889e81de27fd0eaf0a P 6443362bb8e54a9f92661eb1b956df5e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 02:36:17.502108 4129 raft_consensus.cc:2258] T a6f93e3bb66e44889e81de27fd0eaf0a P 6443362bb8e54a9f92661eb1b956df5e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:36:17.502200 14748 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbep4M4/build/debug/bin/kudu with pid 3622
I20260812 02:36:17.502444 4127 raft_consensus.cc:2258] T a6f93e3bb66e44889e81de27fd0eaf0a P c083f474376a432cb7966a18f399e5df [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:36:17.502864 4128 tablet_replica.cc:330] T a6f93e3bb66e44889e81de27fd0eaf0a P cdd90597ed584152abb9301e00df4b77: stopping tablet replica
I20260812 02:36:17.503273 4128 raft_consensus.cc:2229] T a6f93e3bb66e44889e81de27fd0eaf0a P cdd90597ed584152abb9301e00df4b77 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 02:36:17.503640 4128 raft_consensus.cc:2258] T a6f93e3bb66e44889e81de27fd0eaf0a P cdd90597ed584152abb9301e00df4b77 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:36:17.505230 4128 ts_tablet_manager.cc:1905] T a6f93e3bb66e44889e81de27fd0eaf0a P cdd90597ed584152abb9301e00df4b77: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:36:17.508354 4128 ts_tablet_manager.cc:1918] T a6f93e3bb66e44889e81de27fd0eaf0a P cdd90597ed584152abb9301e00df4b77: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 02:36:17.510011 4127 ts_tablet_manager.cc:1905] T a6f93e3bb66e44889e81de27fd0eaf0a P c083f474376a432cb7966a18f399e5df: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:36:17.512955 4127 ts_tablet_manager.cc:1918] T a6f93e3bb66e44889e81de27fd0eaf0a P c083f474376a432cb7966a18f399e5df: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 02:36:17.513175 4127 log.cc:1196] T a6f93e3bb66e44889e81de27fd0eaf0a P c083f474376a432cb7966a18f399e5df: Deleting WAL directory at /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786501898713699-14748-0/minicluster-data/ts-2/wal/wals/a6f93e3bb66e44889e81de27fd0eaf0a
I20260812 02:36:17.513567 4127 ts_tablet_manager.cc:1939] T a6f93e3bb66e44889e81de27fd0eaf0a P c083f474376a432cb7966a18f399e5df: Deleting consensus metadata
I20260812 02:36:17.508558 4128 log.cc:1196] T a6f93e3bb66e44889e81de27fd0eaf0a P cdd90597ed584152abb9301e00df4b77: Deleting WAL directory at /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786501898713699-14748-0/minicluster-data/ts-1/wal/wals/a6f93e3bb66e44889e81de27fd0eaf0a
I20260812 02:36:17.515041 3531 catalog_manager.cc:4841] TS c083f474376a432cb7966a18f399e5df (127.14.103.3:44323): tablet a6f93e3bb66e44889e81de27fd0eaf0a (table test-table [id=99a800d1fc4c4fedad0c56a9588efb03]) successfully deleted
I20260812 02:36:17.515522 4128 ts_tablet_manager.cc:1939] T a6f93e3bb66e44889e81de27fd0eaf0a P cdd90597ed584152abb9301e00df4b77: Deleting consensus metadata
I20260812 02:36:17.516924 3533 catalog_manager.cc:4841] TS cdd90597ed584152abb9301e00df4b77 (127.14.103.2:38647): tablet a6f93e3bb66e44889e81de27fd0eaf0a (table test-table [id=99a800d1fc4c4fedad0c56a9588efb03]) successfully deleted
W20260812 02:36:17.518132 3534 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.103.1:42041 (error 108)
I20260812 02:36:17.519131 14748 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbep4M4/build/debug/bin/kudu with pid 3779
I20260812 02:36:17.521303 4092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:17.519253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.1:42041 (local address 127.0.0.1:57908)
0812 02:36:17.521116 (+ 1863us) negotiation.cc:107] Waiting for socket to connect
0812 02:36:17.521207 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.1:42041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1759}
W20260812 02:36:17.523077 3534 catalog_manager.cc:4584] TS 6443362bb8e54a9f92661eb1b956df5e (127.14.103.1:42041): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a6f93e3bb66e44889e81de27fd0eaf0a: Network error: Client connection negotiation failed: client connection to 127.14.103.1:42041: connect: Connection refused (error 111)
I20260812 02:36:17.528023 14748 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbep4M4/build/debug/bin/kudu with pid 3929
I20260812 02:36:17.536842 14748 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbep4M4/build/debug/bin/kudu with pid 3515
2026-08-12T02:36:17Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1113 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501898713699-14748-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 02:36:17 dist-test-slave-2w3w krb5kdc[4133](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 02:36:17 dist-test-slave-2w3w krb5kdc[4133](info): set up 2 sockets
Aug 12 02:36:17 dist-test-slave-2w3w krb5kdc[4133](info): commencing operation
krb5kdc: starting...
W20260812 02:36:18.700654 14748 mini_kdc.cc:121] Time spent starting KDC: real 1.138s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501898713699-14748-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501898713699-14748-0/krb5kdc/test-user.keytab.
Aug 12 02:36:18 dist-test-slave-2w3w krb5kdc[4133](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502178, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T02:36:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T02:36:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.103.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.103.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.103.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.62.keytab.
Entry for principal kudu/127.14.103.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.62.keytab.
I20260812 02:36:18.842595 14748 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbep4M4/build/debug/bin/kudu
/tmp/dist-test-taskbep4M4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501898713699-14748-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501898713699-14748-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501898713699-14748-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501898713699-14748-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.103.62:42523
--webserver_interface=127.14.103.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.62.keytab
--principal=kudu/127.14.103.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.103.20:36761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.103.62:42523
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501898713699-14748-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501898713699-14748-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501898713699-14748-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:36:18.964179 4149 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:36:18.964491 4149 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:36:18.964533 4149 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:36:18.968780 4149 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:36:18.968858 4149 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:36:18.968879 4149 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:36:18.968896 4149 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:36:18.968912 4149 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 02:36:18.974942 4149 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.103.20:36761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501898713699-14748-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501898713699-14748-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.103.62:42523
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.62.keytab
--principal=kudu/127.14.103.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.103.62:42523
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501898713699-14748-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.103.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4149
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501898713699-14748-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:36:18.976136 4149 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:36:18.977293 4149 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:36:18.986168 4157 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:36:18.986223 4155 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:36:18.986168 4154 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:36:18.986606 4149 server_base.cc:1034] running on GCE node
I20260812 02:36:18.987286 4149 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:36:18.988482 4149 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:36:18.989670 4149 hybrid_clock.cc:648] HybridClock initialized: now 1786502178989629 us; error 54 us; skew 500 ppm
Aug 12 02:36:18 dist-test-slave-2w3w krb5kdc[4133](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502178, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:36:18.992697 4149 init.cc:377] Logged in from keytab as kudu/127.14.103.62@KRBTEST.COM (short username kudu)
I20260812 02:36:18.993726 4149 webserver.cc:460] Webserver started at http://127.14.103.62:43933/ using document root <none> and password file <none>
I20260812 02:36:18.994262 4149 fs_manager.cc:359] Metadata directory not provided
I20260812 02:36:18.994328 4149 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:36:18.994498 4149 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:36:18.996635 4149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501898713699-14748-0/minicluster-data/master-0/data/instance:
uuid: "cfca43c27bef497f88696682fae47b1f"
format_stamp: "Formatted at 2026-08-12 02:36:18 on dist-test-slave-2w3w"
server_key: "406cef35c390576a23ffbc830d6486ce"
server_key_iv: "084605669d842b13f460f82ffd4465e0"
server_key_version: "encryptionkey@0"
I20260812 02:36:18.997181 4149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501898713699-14748-0/minicluster-data/master-0/wal/instance:
uuid: "cfca43c27bef497f88696682fae47b1f"
format_stamp: "Formatted at 2026-08-12 02:36:18 on dist-test-slave-2w3w"
server_key: "406cef35c390576a23ffbc830d6486ce"
server_key_iv: "084605669d842b13f460f82ffd4465e0"
server_key_version: "encryptionkey@0"
I20260812 02:36:19.001067 4149 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260812 02:36:19.003726 4165 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:36:19.005366 4149 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 02:36:19.005556 4149 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501898713699-14748-0/minicluster-data/master-0/data,/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501898713699-14748-0/minicluster-data/master-0/wal
uuid: "cfca43c27bef497f88696682fae47b1f"
format_stamp: "Formatted at 2026-08-12 02:36:18 on dist-test-slave-2w3w"
server_key: "406cef35c390576a23ffbc830d6486ce"
server_key_iv: "084605669d842b13f460f82ffd4465e0"
server_key_version: "encryptionkey@0"
I20260812 02:36:19.005697 4149 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501898713699-14748-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501898713699-14748-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501898713699-14748-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:36:19.030702 4149 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:36:19.034034 4149 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:36:19.034266 4149 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:36:19.042342 4149 rpc_server.cc:307] RPC server started. Bound to: 127.14.103.62:42523
I20260812 02:36:19.042368 4229 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.103.62:42523 every 8 connection(s)
I20260812 02:36:19.043402 4149 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501898713699-14748-0/minicluster-data/master-0/data/info.pb
I20260812 02:36:19.044633 14748 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbep4M4/build/debug/bin/kudu as pid 4149
I20260812 02:36:19.044799 14748 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501898713699-14748-0/minicluster-data/master-0/wal/instance
I20260812 02:36:19.045156 14748 external_mini_cluster.cc:1428] Setting key 6a46c51fe9ba7d4009d596a9274eace4
I20260812 02:36:19.047065 4230 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:36:19.055182 4230 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cfca43c27bef497f88696682fae47b1f: Bootstrap starting.
Aug 12 02:36:19 dist-test-slave-2w3w krb5kdc[4133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502178, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.103.62@KRBTEST.COM
I20260812 02:36:19.058331 4230 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cfca43c27bef497f88696682fae47b1f: Neither blocks nor log segments found. Creating new log.
I20260812 02:36:19.059283 4230 log.cc:824] T 00000000000000000000000000000000 P cfca43c27bef497f88696682fae47b1f: Log is configured to *not* fsync() on all Append() calls
I20260812 02:36:19.061584 4230 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cfca43c27bef497f88696682fae47b1f: No bootstrap required, opened a new log
I20260812 02:36:19.064486 4234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:19.047195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58406 (local address 127.14.103.62:42523)
0812 02:36:19.049564 (+ 2369us) server_negotiation.cc:206] Beginning negotiation
0812 02:36:19.049598 (+ 34us) server_negotiation.cc:399] Waiting for connection header
0812 02:36:19.049647 (+ 49us) server_negotiation.cc:407] Connection header received
0812 02:36:19.050793 (+ 1146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:36:19.050820 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:36:19.051310 (+ 490us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:36:19.051734 (+ 424us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:36:19.052690 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:19.053711 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:19.055051 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:19.055624 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:19.055748 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:36:19.058868 (+ 3120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:36:19.058873 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:36:19.058888 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:36:19.058919 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:36:19.060835 (+ 1916us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:36:19.061326 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:36:19.061331 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:36:19.061335 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:36:19.061474 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:36:19.061789 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:36:19.061793 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:36:19.061795 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:36:19.062417 (+ 622us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:36:19.062648 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0812 02:36:19.062855 (+ 207us) server_negotiation.cc:299] Negotiation successful
0812 02:36:19.063229 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2176,"thread_start_us":172,"threads_started":1}
I20260812 02:36:19.066660 4230 raft_consensus.cc:348] T 00000000000000000000000000000000 P cfca43c27bef497f88696682fae47b1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfca43c27bef497f88696682fae47b1f" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 42523 } }
I20260812 02:36:19.067010 4230 raft_consensus.cc:374] T 00000000000000000000000000000000 P cfca43c27bef497f88696682fae47b1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:36:19.067078 4230 raft_consensus.cc:729] T 00000000000000000000000000000000 P cfca43c27bef497f88696682fae47b1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfca43c27bef497f88696682fae47b1f, State: Initialized, Role: FOLLOWER
I20260812 02:36:19.067497 4230 consensus_queue.cc:260] T 00000000000000000000000000000000 P cfca43c27bef497f88696682fae47b1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfca43c27bef497f88696682fae47b1f" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 42523 } }
I20260812 02:36:19.067628 4230 raft_consensus.cc:388] T 00000000000000000000000000000000 P cfca43c27bef497f88696682fae47b1f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 02:36:19.067708 4230 raft_consensus.cc:482] T 00000000000000000000000000000000 P cfca43c27bef497f88696682fae47b1f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 02:36:19.067813 4230 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cfca43c27bef497f88696682fae47b1f [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:36:19.068878 4230 raft_consensus.cc:504] T 00000000000000000000000000000000 P cfca43c27bef497f88696682fae47b1f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfca43c27bef497f88696682fae47b1f" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 42523 } }
I20260812 02:36:19.069252 4230 leader_election.cc:302] T 00000000000000000000000000000000 P cfca43c27bef497f88696682fae47b1f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cfca43c27bef497f88696682fae47b1f; no voters:
I20260812 02:36:19.069588 4230 leader_election.cc:288] T 00000000000000000000000000000000 P cfca43c27bef497f88696682fae47b1f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 02:36:19.069859 4236 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cfca43c27bef497f88696682fae47b1f [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:36:19.070127 4236 raft_consensus.cc:686] T 00000000000000000000000000000000 P cfca43c27bef497f88696682fae47b1f [term 1 LEADER]: Becoming Leader. State: Replica: cfca43c27bef497f88696682fae47b1f, State: Running, Role: LEADER
I20260812 02:36:19.070446 4236 consensus_queue.cc:237] T 00000000000000000000000000000000 P cfca43c27bef497f88696682fae47b1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfca43c27bef497f88696682fae47b1f" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 42523 } }
I20260812 02:36:19.070921 4230 sys_catalog.cc:564] T 00000000000000000000000000000000 P cfca43c27bef497f88696682fae47b1f [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:36:19.072086 4238 sys_catalog.cc:455] T 00000000000000000000000000000000 P cfca43c27bef497f88696682fae47b1f [sys.catalog]: SysCatalogTable state changed. Reason: New leader cfca43c27bef497f88696682fae47b1f. Latest consensus state: current_term: 1 leader_uuid: "cfca43c27bef497f88696682fae47b1f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfca43c27bef497f88696682fae47b1f" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 42523 } } }
I20260812 02:36:19.072227 4238 sys_catalog.cc:458] T 00000000000000000000000000000000 P cfca43c27bef497f88696682fae47b1f [sys.catalog]: This master's current role is: LEADER
I20260812 02:36:19.072568 4237 sys_catalog.cc:455] T 00000000000000000000000000000000 P cfca43c27bef497f88696682fae47b1f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cfca43c27bef497f88696682fae47b1f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfca43c27bef497f88696682fae47b1f" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 42523 } } }
I20260812 02:36:19.072680 4237 sys_catalog.cc:458] T 00000000000000000000000000000000 P cfca43c27bef497f88696682fae47b1f [sys.catalog]: This master's current role is: LEADER
W20260812 02:36:19.076097 4251 catalog_manager.cc:1558] T 00000000000000000000000000000000 P cfca43c27bef497f88696682fae47b1f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 02:36:19.076191 4251 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 02:36:19.076272 4245 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 02:36:19.077091 4245 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 02:36:19.083230 4245 catalog_manager.cc:1347] Generated new cluster ID: 5ce9b0f885dd489ba2c02113d942aed4
I20260812 02:36:19.083317 4245 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 02:36:19.106658 4245 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 02:36:19.108021 4245 catalog_manager.cc:1504] Loading token signing keys...
I20260812 02:36:19.121543 4245 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cfca43c27bef497f88696682fae47b1f: Generated new TSK 0
I20260812 02:36:19.122375 4245 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.103.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.103.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.103.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.1.keytab.
Entry for principal kudu/127.14.103.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.1.keytab.
I20260812 02:36:19.164000 14748 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbep4M4/build/debug/bin/kudu
/tmp/dist-test-taskbep4M4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501898713699-14748-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501898713699-14748-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501898713699-14748-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501898713699-14748-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.103.1:0
--local_ip_for_outbound_sockets=127.14.103.1
--webserver_interface=127.14.103.1
--webserver_port=0
--tserver_master_addrs=127.14.103.62:42523
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.1.keytab
--principal=kudu/127.14.103.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.103.20:36761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501898713699-14748-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501898713699-14748-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501898713699-14748-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:36:19.280974 4257 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:36:19.281256 4257 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:36:19.281304 4257 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:36:19.285681 4257 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:36:19.285775 4257 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:36:19.285929 4257 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.103.1
I20260812 02:36:19.291754 4257 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.103.20:36761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501898713699-14748-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501898713699-14748-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.1.keytab
--principal=kudu/127.14.103.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.103.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501898713699-14748-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.103.1
--webserver_port=0
--tserver_master_addrs=127.14.103.62:42523
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4257
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.103.1
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501898713699-14748-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:36:19.292915 4257 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:36:19.293962 4257 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:36:19.303756 4264 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:36:19.303755 4266 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:36:19.303961 4257 server_base.cc:1034] running on GCE node
W20260812 02:36:19.304008 4263 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:36:19.304622 4257 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:36:19.305339 4257 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:36:19.306524 4257 hybrid_clock.cc:648] HybridClock initialized: now 1786502179306498 us; error 42 us; skew 500 ppm
Aug 12 02:36:19 dist-test-slave-2w3w krb5kdc[4133](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502179, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:36:19.309518 4257 init.cc:377] Logged in from keytab as kudu/127.14.103.1@KRBTEST.COM (short username kudu)
I20260812 02:36:19.310518 4257 webserver.cc:460] Webserver started at http://127.14.103.1:37399/ using document root <none> and password file <none>
I20260812 02:36:19.311158 4257 fs_manager.cc:359] Metadata directory not provided
I20260812 02:36:19.311223 4257 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:36:19.311435 4257 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:36:19.313517 4257 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501898713699-14748-0/minicluster-data/ts-0/data/instance:
uuid: "e39c748b3f8e47b3842e7092fecee47f"
format_stamp: "Formatted at 2026-08-12 02:36:19 on dist-test-slave-2w3w"
server_key: "5db9490562bfd1119794e25143abec1f"
server_key_iv: "b3acfc83936792c9a72cd7ca6664451e"
server_key_version: "encryptionkey@0"
I20260812 02:36:19.314090 4257 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501898713699-14748-0/minicluster-data/ts-0/wal/instance:
uuid: "e39c748b3f8e47b3842e7092fecee47f"
format_stamp: "Formatted at 2026-08-12 02:36:19 on dist-test-slave-2w3w"
server_key: "5db9490562bfd1119794e25143abec1f"
server_key_iv: "b3acfc83936792c9a72cd7ca6664451e"
server_key_version: "encryptionkey@0"
I20260812 02:36:19.318034 4257 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260812 02:36:19.320535 4274 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:36:19.321846 4257 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260812 02:36:19.321981 4257 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501898713699-14748-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501898713699-14748-0/minicluster-data/ts-0/wal
uuid: "e39c748b3f8e47b3842e7092fecee47f"
format_stamp: "Formatted at 2026-08-12 02:36:19 on dist-test-slave-2w3w"
server_key: "5db9490562bfd1119794e25143abec1f"
server_key_iv: "b3acfc83936792c9a72cd7ca6664451e"
server_key_version: "encryptionkey@0"
I20260812 02:36:19.322101 4257 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501898713699-14748-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501898713699-14748-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501898713699-14748-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:36:19.343047 4257 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:36:19.346343 4257 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:36:19.346570 4257 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:36:19.347256 4257 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:36:19.348136 4257 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:36:19.348217 4257 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:36:19.348304 4257 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:36:19.348354 4257 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:36:19.357393 4257 rpc_server.cc:307] RPC server started. Bound to: 127.14.103.1:37793
I20260812 02:36:19.357420 4407 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.103.1:37793 every 8 connection(s)
I20260812 02:36:19.358296 4257 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501898713699-14748-0/minicluster-data/ts-0/data/info.pb
I20260812 02:36:19.362102 14748 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbep4M4/build/debug/bin/kudu as pid 4257
I20260812 02:36:19.362252 14748 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501898713699-14748-0/minicluster-data/ts-0/wal/instance
I20260812 02:36:19.362573 14748 external_mini_cluster.cc:1428] Setting key 7793632f4895fb3bbdbec87b6981c635
Aug 12 02:36:19 dist-test-slave-2w3w krb5kdc[4133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502179, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.1@KRBTEST.COM for kudu/127.14.103.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.14.103.2@KRBTEST.COM; defaulting to no policy
I20260812 02:36:19.374595 4234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:19.360164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.1:33541 (local address 127.14.103.62:42523)
0812 02:36:19.360328 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0812 02:36:19.360333 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:36:19.361295 (+ 962us) server_negotiation.cc:407] Connection header received
0812 02:36:19.362931 (+ 1636us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:36:19.362935 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:36:19.363030 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:36:19.363122 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:36:19.364737 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:19.365572 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:19.366890 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:19.367347 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:19.367480 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:36:19.370061 (+ 2581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:36:19.370066 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:36:19.370071 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:36:19.370108 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:36:19.372095 (+ 1987us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:36:19.372769 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:36:19.372773 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:36:19.372775 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:36:19.372893 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:36:19.373265 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:36:19.373271 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:36:19.373276 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:36:19.373547 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:36:19.373676 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0812 02:36:19.374291 (+ 615us) server_negotiation.cc:299] Negotiation successful
0812 02:36:19.374447 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260812 02:36:19.375613 4411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:19.360590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:42523 (local address 127.14.103.1:33541)
0812 02:36:19.361102 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0812 02:36:19.361163 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0812 02:36:19.362679 (+ 1516us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:36:19.363361 (+ 682us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:36:19.363368 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:36:19.363909 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:36:19.364562 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:36:19.364578 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:36:19.365714 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:36:19.365716 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:36:19.366725 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:36:19.366735 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:36:19.367514 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:36:19.367518 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:36:19.367667 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:36:19.368333 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:36:19.368350 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:36:19.369874 (+ 1524us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:36:19.372245 (+ 2371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:36:19.372250 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:36:19.372265 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:36:19.372650 (+ 385us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:36:19.373006 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:36:19.373009 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:36:19.373010 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:36:19.373161 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:36:19.373699 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:36:19.373704 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:36:19.373994 (+ 290us) client_negotiation.cc:769] Sending connection context
0812 02:36:19.374220 (+ 226us) client_negotiation.cc:240] Negotiation successful
0812 02:36:19.374533 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":131,"threads_started":1}
I20260812 02:36:19.377104 4409 heartbeater.cc:347] Connected to a master server at 127.14.103.62:42523
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.103.2@KRBTEST.COM" created.
I20260812 02:36:19.377381 4409 heartbeater.cc:464] Registering TS with master...
I20260812 02:36:19.377946 4409 heartbeater.cc:511] Master 127.14.103.62:42523 requested a full tablet report, sending...
I20260812 02:36:19.379611 4185 ts_manager.cc:194] Registered new tserver with Master: e39c748b3f8e47b3842e7092fecee47f (127.14.103.1:37793)
I20260812 02:36:19.380762 4185 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.103.1@KRBTEST.COM'} at 127.14.103.1:33541
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.103.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.2.keytab.
Entry for principal kudu/127.14.103.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.2.keytab.
I20260812 02:36:19.394536 14748 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbep4M4/build/debug/bin/kudu
/tmp/dist-test-taskbep4M4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501898713699-14748-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501898713699-14748-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501898713699-14748-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501898713699-14748-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.103.2:0
--local_ip_for_outbound_sockets=127.14.103.2
--webserver_interface=127.14.103.2
--webserver_port=0
--tserver_master_addrs=127.14.103.62:42523
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.2.keytab
--principal=kudu/127.14.103.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.103.20:36761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501898713699-14748-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501898713699-14748-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501898713699-14748-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:36:19.512317 4414 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:36:19.512600 4414 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:36:19.512673 4414 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:36:19.516904 4414 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:36:19.516975 4414 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:36:19.517071 4414 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.103.2
I20260812 02:36:19.522635 4414 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.103.20:36761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501898713699-14748-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501898713699-14748-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.2.keytab
--principal=kudu/127.14.103.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.103.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501898713699-14748-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.103.2
--webserver_port=0
--tserver_master_addrs=127.14.103.62:42523
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4414
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.103.2
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501898713699-14748-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:36:19.523815 4414 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:36:19.524855 4414 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:36:19.534592 4424 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:36:19.534638 4427 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:36:19.534632 4414 server_base.cc:1034] running on GCE node
W20260812 02:36:19.534592 4425 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:36:19.535384 4414 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:36:19.535974 4414 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:36:19.537158 4414 hybrid_clock.cc:648] HybridClock initialized: now 1786502179537130 us; error 45 us; skew 500 ppm
Aug 12 02:36:19 dist-test-slave-2w3w krb5kdc[4133](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502179, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:36:19.540205 4414 init.cc:377] Logged in from keytab as kudu/127.14.103.2@KRBTEST.COM (short username kudu)
I20260812 02:36:19.541086 4414 webserver.cc:460] Webserver started at http://127.14.103.2:45195/ using document root <none> and password file <none>
I20260812 02:36:19.541628 4414 fs_manager.cc:359] Metadata directory not provided
I20260812 02:36:19.541690 4414 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:36:19.541863 4414 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:36:19.544083 4414 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501898713699-14748-0/minicluster-data/ts-1/data/instance:
uuid: "bbea1665b24c4bf39ba99ca7696d1462"
format_stamp: "Formatted at 2026-08-12 02:36:19 on dist-test-slave-2w3w"
server_key: "82ba40e79a1bf66809c6d0423ea67841"
server_key_iv: "65b990ab7180c6523482147743797441"
server_key_version: "encryptionkey@0"
I20260812 02:36:19.544672 4414 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501898713699-14748-0/minicluster-data/ts-1/wal/instance:
uuid: "bbea1665b24c4bf39ba99ca7696d1462"
format_stamp: "Formatted at 2026-08-12 02:36:19 on dist-test-slave-2w3w"
server_key: "82ba40e79a1bf66809c6d0423ea67841"
server_key_iv: "65b990ab7180c6523482147743797441"
server_key_version: "encryptionkey@0"
I20260812 02:36:19.548647 4414 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260812 02:36:19.551187 4434 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:36:19.552443 4414 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260812 02:36:19.552570 4414 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501898713699-14748-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501898713699-14748-0/minicluster-data/ts-1/wal
uuid: "bbea1665b24c4bf39ba99ca7696d1462"
format_stamp: "Formatted at 2026-08-12 02:36:19 on dist-test-slave-2w3w"
server_key: "82ba40e79a1bf66809c6d0423ea67841"
server_key_iv: "65b990ab7180c6523482147743797441"
server_key_version: "encryptionkey@0"
I20260812 02:36:19.552681 4414 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501898713699-14748-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501898713699-14748-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501898713699-14748-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:36:19.573937 4414 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:36:19.577477 4414 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:36:19.577696 4414 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:36:19.578414 4414 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:36:19.579373 4414 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:36:19.579442 4414 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:36:19.579512 4414 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:36:19.579557 4414 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:36:19.589318 4414 rpc_server.cc:307] RPC server started. Bound to: 127.14.103.2:33357
I20260812 02:36:19.589390 4577 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.103.2:33357 every 8 connection(s)
I20260812 02:36:19.590255 4414 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501898713699-14748-0/minicluster-data/ts-1/data/info.pb
I20260812 02:36:19.593025 14748 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbep4M4/build/debug/bin/kudu as pid 4414
I20260812 02:36:19.593160 14748 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501898713699-14748-0/minicluster-data/ts-1/wal/instance
I20260812 02:36:19.593482 14748 external_mini_cluster.cc:1428] Setting key a8906acdb031dc4223ecfa68148c526b
Aug 12 02:36:19 dist-test-slave-2w3w krb5kdc[4133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502179, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.2@KRBTEST.COM for kudu/127.14.103.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.14.103.3@KRBTEST.COM; defaulting to no policy
I20260812 02:36:19.605039 4234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:19.592211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.2:51801 (local address 127.14.103.62:42523)
0812 02:36:19.592383 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0812 02:36:19.592389 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 02:36:19.593372 (+ 983us) server_negotiation.cc:407] Connection header received
0812 02:36:19.595120 (+ 1748us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:36:19.595123 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:36:19.595175 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:36:19.595251 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:36:19.596461 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:19.597002 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:19.598158 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:19.598395 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:19.598507 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:36:19.601263 (+ 2756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:36:19.601266 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:36:19.601268 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:36:19.601293 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:36:19.602785 (+ 1492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:36:19.603410 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:36:19.603413 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:36:19.603415 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:36:19.603496 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:36:19.603878 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:36:19.603881 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:36:19.603883 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:36:19.604079 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:36:19.604189 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0812 02:36:19.604723 (+ 534us) server_negotiation.cc:299] Negotiation successful
0812 02:36:19.604869 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260812 02:36:19.605908 4581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:19.592585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:42523 (local address 127.14.103.2:51801)
0812 02:36:19.593171 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0812 02:36:19.593240 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0812 02:36:19.594872 (+ 1632us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:36:19.595429 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:36:19.595436 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:36:19.595802 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:36:19.596309 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:36:19.596326 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:36:19.597130 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:36:19.597133 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:36:19.598028 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:36:19.598038 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:36:19.598553 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:36:19.598557 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:36:19.598730 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:36:19.599363 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:36:19.599380 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:36:19.600962 (+ 1582us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:36:19.602945 (+ 1983us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:36:19.602952 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:36:19.603007 (+ 55us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:36:19.603282 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:36:19.603596 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:36:19.603599 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:36:19.603601 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:36:19.603766 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:36:19.604189 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:36:19.604198 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:36:19.604469 (+ 271us) client_negotiation.cc:769] Sending connection context
0812 02:36:19.604660 (+ 191us) client_negotiation.cc:240] Negotiation successful
0812 02:36:19.605052 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":181,"threads_started":1}
I20260812 02:36:19.607475 4578 heartbeater.cc:347] Connected to a master server at 127.14.103.62:42523
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.103.3@KRBTEST.COM" created.
I20260812 02:36:19.607755 4578 heartbeater.cc:464] Registering TS with master...
I20260812 02:36:19.608436 4578 heartbeater.cc:511] Master 127.14.103.62:42523 requested a full tablet report, sending...
I20260812 02:36:19.609781 4185 ts_manager.cc:194] Registered new tserver with Master: bbea1665b24c4bf39ba99ca7696d1462 (127.14.103.2:33357)
I20260812 02:36:19.610473 4185 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.103.2@KRBTEST.COM'} at 127.14.103.2:51801
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.103.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.3.keytab.
Entry for principal kudu/127.14.103.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.3.keytab.
I20260812 02:36:19.627214 14748 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbep4M4/build/debug/bin/kudu
/tmp/dist-test-taskbep4M4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501898713699-14748-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501898713699-14748-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501898713699-14748-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501898713699-14748-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.103.3:0
--local_ip_for_outbound_sockets=127.14.103.3
--webserver_interface=127.14.103.3
--webserver_port=0
--tserver_master_addrs=127.14.103.62:42523
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.3.keytab
--principal=kudu/127.14.103.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.103.20:36761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501898713699-14748-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501898713699-14748-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501898713699-14748-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:36:19.739529 4584 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:36:19.739812 4584 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:36:19.739866 4584 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:36:19.743942 4584 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:36:19.744009 4584 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:36:19.744069 4584 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.103.3
I20260812 02:36:19.749430 4584 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.103.20:36761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501898713699-14748-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501898713699-14748-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.3.keytab
--principal=kudu/127.14.103.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.103.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501898713699-14748-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.103.3
--webserver_port=0
--tserver_master_addrs=127.14.103.62:42523
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4584
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.103.3
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501898713699-14748-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:36:19.750530 4584 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:36:19.751613 4584 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:36:19.762140 4590 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:36:19.762142 4589 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:36:19.762355 4584 server_base.cc:1034] running on GCE node
W20260812 02:36:19.762140 4592 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:36:19.763121 4584 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:36:19.763850 4584 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:36:19.765066 4584 hybrid_clock.cc:648] HybridClock initialized: now 1786502179765046 us; error 35 us; skew 500 ppm
Aug 12 02:36:19 dist-test-slave-2w3w krb5kdc[4133](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502179, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:36:19.768239 4584 init.cc:377] Logged in from keytab as kudu/127.14.103.3@KRBTEST.COM (short username kudu)
I20260812 02:36:19.769215 4584 webserver.cc:460] Webserver started at http://127.14.103.3:42225/ using document root <none> and password file <none>
I20260812 02:36:19.769804 4584 fs_manager.cc:359] Metadata directory not provided
I20260812 02:36:19.769872 4584 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:36:19.770058 4584 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:36:19.772544 4584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501898713699-14748-0/minicluster-data/ts-2/data/instance:
uuid: "a9dc9ad78d1244baba20193904ca79ac"
format_stamp: "Formatted at 2026-08-12 02:36:19 on dist-test-slave-2w3w"
server_key: "7bf1f5d47ad230da04fb388e3bcb23a2"
server_key_iv: "89e5673f01cf569ee7bbef37803ca930"
server_key_version: "encryptionkey@0"
I20260812 02:36:19.773131 4584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501898713699-14748-0/minicluster-data/ts-2/wal/instance:
uuid: "a9dc9ad78d1244baba20193904ca79ac"
format_stamp: "Formatted at 2026-08-12 02:36:19 on dist-test-slave-2w3w"
server_key: "7bf1f5d47ad230da04fb388e3bcb23a2"
server_key_iv: "89e5673f01cf569ee7bbef37803ca930"
server_key_version: "encryptionkey@0"
I20260812 02:36:19.777447 4584 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260812 02:36:19.780373 4601 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:36:19.781787 4584 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260812 02:36:19.781921 4584 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501898713699-14748-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501898713699-14748-0/minicluster-data/ts-2/wal
uuid: "a9dc9ad78d1244baba20193904ca79ac"
format_stamp: "Formatted at 2026-08-12 02:36:19 on dist-test-slave-2w3w"
server_key: "7bf1f5d47ad230da04fb388e3bcb23a2"
server_key_iv: "89e5673f01cf569ee7bbef37803ca930"
server_key_version: "encryptionkey@0"
I20260812 02:36:19.782032 4584 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501898713699-14748-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501898713699-14748-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501898713699-14748-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:36:19.812239 4584 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:36:19.816036 4584 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:36:19.816231 4584 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:36:19.816886 4584 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:36:19.817821 4584 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:36:19.817878 4584 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:36:19.817926 4584 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:36:19.817943 4584 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:36:19.828429 4584 rpc_server.cc:307] RPC server started. Bound to: 127.14.103.3:34831
I20260812 02:36:19.828490 4730 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.103.3:34831 every 8 connection(s)
I20260812 02:36:19.829418 4584 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501898713699-14748-0/minicluster-data/ts-2/data/info.pb
I20260812 02:36:19.835420 14748 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbep4M4/build/debug/bin/kudu as pid 4584
I20260812 02:36:19.835587 14748 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501898713699-14748-0/minicluster-data/ts-2/wal/instance
I20260812 02:36:19.835881 14748 external_mini_cluster.cc:1428] Setting key 51dbdffe50f81af02ed112a411e10988
Aug 12 02:36:19 dist-test-slave-2w3w krb5kdc[4133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502179, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.3@KRBTEST.COM for kudu/127.14.103.62@KRBTEST.COM
I20260812 02:36:19.844228 4234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:19.831368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.3:49293 (local address 127.14.103.62:42523)
0812 02:36:19.831524 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0812 02:36:19.831528 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:36:19.832542 (+ 1014us) server_negotiation.cc:407] Connection header received
0812 02:36:19.833867 (+ 1325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:36:19.833870 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:36:19.833917 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:36:19.833998 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:36:19.835414 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:19.836200 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:19.837412 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:19.837634 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:19.837747 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:36:19.840410 (+ 2663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:36:19.840414 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:36:19.840416 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:36:19.840444 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:36:19.841998 (+ 1554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:36:19.842578 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:36:19.842582 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:36:19.842583 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:36:19.842664 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:36:19.843071 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:36:19.843075 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:36:19.843076 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:36:19.843280 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:36:19.843393 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0812 02:36:19.843922 (+ 529us) server_negotiation.cc:299] Negotiation successful
0812 02:36:19.844060 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260812 02:36:19.845112 4733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:19.831733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:42523 (local address 127.14.103.3:49293)
0812 02:36:19.832335 (+ 602us) negotiation.cc:107] Waiting for socket to connect
0812 02:36:19.832381 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0812 02:36:19.833674 (+ 1293us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:36:19.834171 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:36:19.834178 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:36:19.834549 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:36:19.835211 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:36:19.835236 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:36:19.836358 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:36:19.836361 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:36:19.837270 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:36:19.837280 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:36:19.837792 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:36:19.837797 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:36:19.837993 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:36:19.838557 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:36:19.838575 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:36:19.840175 (+ 1600us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:36:19.842137 (+ 1962us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:36:19.842144 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:36:19.842163 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:36:19.842456 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:36:19.842755 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:36:19.842760 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:36:19.842762 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:36:19.842916 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:36:19.843390 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:36:19.843398 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:36:19.843668 (+ 270us) client_negotiation.cc:769] Sending connection context
0812 02:36:19.843861 (+ 193us) client_negotiation.cc:240] Negotiation successful
0812 02:36:19.844233 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":132,"threads_started":1}
I20260812 02:36:19.846269 4731 heartbeater.cc:347] Connected to a master server at 127.14.103.62:42523
I20260812 02:36:19.846539 4731 heartbeater.cc:464] Registering TS with master...
I20260812 02:36:19.847136 4731 heartbeater.cc:511] Master 127.14.103.62:42523 requested a full tablet report, sending...
I20260812 02:36:19.848201 4185 ts_manager.cc:194] Registered new tserver with Master: a9dc9ad78d1244baba20193904ca79ac (127.14.103.3:34831)
I20260812 02:36:19.848996 4185 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.103.3@KRBTEST.COM'} at 127.14.103.3:49293
I20260812 02:36:19.849654 14748 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 02:36:19.859849 4234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:19.851687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58412 (local address 127.14.103.62:42523)
0812 02:36:19.851839 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0812 02:36:19.851844 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:36:19.852066 (+ 222us) server_negotiation.cc:407] Connection header received
0812 02:36:19.852222 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:36:19.852226 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:36:19.852277 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:36:19.852355 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:36:19.853090 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:19.853628 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:19.854498 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:19.854732 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:19.854851 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:36:19.856411 (+ 1560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:36:19.856415 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:36:19.856417 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:36:19.856447 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:36:19.857984 (+ 1537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:36:19.858430 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:36:19.858434 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:36:19.858436 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:36:19.858520 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:36:19.858800 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:36:19.858804 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:36:19.858805 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:36:19.859132 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:36:19.859272 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0812 02:36:19.859529 (+ 257us) server_negotiation.cc:299] Negotiation successful
0812 02:36:19.859652 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260812 02:36:19.863040 4183 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58412:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 02:36:19.866025 4183 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 02:36:19.882170 4742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:19.876030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.1:37793 (local address 127.0.0.1:53076)
0812 02:36:19.876764 (+ 734us) negotiation.cc:107] Waiting for socket to connect
0812 02:36:19.876780 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 02:36:19.876887 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:36:19.877555 (+ 668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:36:19.877559 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:36:19.877588 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:36:19.877827 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:36:19.877836 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:36:19.879192 (+ 1356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:36:19.879195 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:36:19.880561 (+ 1366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:36:19.880571 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:36:19.881412 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:36:19.881415 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:36:19.881530 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:36:19.881582 (+ 52us) client_negotiation.cc:769] Sending connection context
0812 02:36:19.881845 (+ 263us) client_negotiation.cc:240] Negotiation successful
0812 02:36:19.881994 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":650,"mutex_wait_us":48,"spinlock_wait_cycles":3712,"thread_start_us":112,"threads_started":1}
I20260812 02:36:19.882567 4744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:19.875573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.2:33357 (local address 127.0.0.1:34254)
0812 02:36:19.876440 (+ 867us) negotiation.cc:107] Waiting for socket to connect
0812 02:36:19.876505 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0812 02:36:19.876657 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:36:19.877211 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:36:19.877217 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:36:19.877239 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:36:19.877394 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:36:19.877401 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:36:19.878950 (+ 1549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:36:19.878954 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:36:19.880186 (+ 1232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:36:19.880199 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:36:19.881639 (+ 1440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:36:19.881643 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:36:19.882224 (+ 581us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:36:19.882251 (+ 27us) client_negotiation.cc:769] Sending connection context
0812 02:36:19.882338 (+ 87us) client_negotiation.cc:240] Negotiation successful
0812 02:36:19.882413 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":805,"mutex_wait_us":59,"spinlock_wait_cycles":8576,"thread_start_us":102,"threads_started":1}
I20260812 02:36:19.882679 4748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:19.876206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53076 (local address 127.14.103.1:37793)
0812 02:36:19.876621 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0812 02:36:19.876636 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0812 02:36:19.876883 (+ 247us) server_negotiation.cc:407] Connection header received
0812 02:36:19.877000 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:36:19.877008 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:36:19.877206 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:36:19.877390 (+ 184us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:36:19.877972 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:19.879063 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:19.880692 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:19.881274 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:19.881398 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:36:19.881560 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0812 02:36:19.881885 (+ 325us) server_negotiation.cc:299] Negotiation successful
0812 02:36:19.881996 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":138,"threads_started":1}
I20260812 02:36:19.883199 4747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:19.876057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34254 (local address 127.14.103.2:33357)
0812 02:36:19.876570 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0812 02:36:19.876583 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0812 02:36:19.876609 (+ 26us) server_negotiation.cc:407] Connection header received
0812 02:36:19.876853 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:36:19.876860 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:36:19.877062 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:36:19.877223 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:36:19.877520 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:19.878812 (+ 1292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:19.880344 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:19.881310 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:19.881432 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:36:19.881573 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0812 02:36:19.882884 (+ 1311us) server_negotiation.cc:299] Negotiation successful
0812 02:36:19.883026 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":398,"thread_start_us":120,"threads_started":1}
I20260812 02:36:19.883541 4745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:19.875262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.3:34831 (local address 127.0.0.1:35628)
0812 02:36:19.876440 (+ 1178us) negotiation.cc:107] Waiting for socket to connect
0812 02:36:19.876505 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0812 02:36:19.876657 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:36:19.877555 (+ 898us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:36:19.877558 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:36:19.877584 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:36:19.877841 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:36:19.877847 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:36:19.880478 (+ 2631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:36:19.880482 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:36:19.881707 (+ 1225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:36:19.881717 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:36:19.883137 (+ 1420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:36:19.883140 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:36:19.883243 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:36:19.883264 (+ 21us) client_negotiation.cc:769] Sending connection context
0812 02:36:19.883341 (+ 77us) client_negotiation.cc:240] Negotiation successful
0812 02:36:19.883409 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":968,"mutex_wait_us":85,"spinlock_wait_cycles":7296,"thread_start_us":105,"threads_started":1}
I20260812 02:36:19.883961 4743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:19.875404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35628 (local address 127.14.103.3:34831)
0812 02:36:19.876984 (+ 1580us) server_negotiation.cc:206] Beginning negotiation
0812 02:36:19.876997 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0812 02:36:19.877017 (+ 20us) server_negotiation.cc:407] Connection header received
0812 02:36:19.877086 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:36:19.877091 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:36:19.877247 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:36:19.877389 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:36:19.879056 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:19.880315 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:19.881876 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:19.882860 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:19.882998 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:36:19.883149 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0812 02:36:19.883769 (+ 620us) server_negotiation.cc:299] Negotiation successful
0812 02:36:19.883852 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1464,"thread_start_us":191,"threads_started":1}
I20260812 02:36:19.885962 4492 tablet_service.cc:1467] Processing CreateTablet for tablet ed5a63e0fdab476e95574f5860d8bf26 (DEFAULT_TABLE table=test-table [id=29f5f55bd7584e6787d8bc5615102620]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:36:19.886023 4328 tablet_service.cc:1467] Processing CreateTablet for tablet ed5a63e0fdab476e95574f5860d8bf26 (DEFAULT_TABLE table=test-table [id=29f5f55bd7584e6787d8bc5615102620]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:36:19.886670 4659 tablet_service.cc:1467] Processing CreateTablet for tablet ed5a63e0fdab476e95574f5860d8bf26 (DEFAULT_TABLE table=test-table [id=29f5f55bd7584e6787d8bc5615102620]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:36:19.887233 4328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed5a63e0fdab476e95574f5860d8bf26. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:36:19.887240 4492 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed5a63e0fdab476e95574f5860d8bf26. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:36:19.887749 4659 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed5a63e0fdab476e95574f5860d8bf26. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:36:19.893560 4749 tablet_bootstrap.cc:492] T ed5a63e0fdab476e95574f5860d8bf26 P bbea1665b24c4bf39ba99ca7696d1462: Bootstrap starting.
I20260812 02:36:19.895548 4750 tablet_bootstrap.cc:492] T ed5a63e0fdab476e95574f5860d8bf26 P e39c748b3f8e47b3842e7092fecee47f: Bootstrap starting.
I20260812 02:36:19.895126 4751 tablet_bootstrap.cc:492] T ed5a63e0fdab476e95574f5860d8bf26 P a9dc9ad78d1244baba20193904ca79ac: Bootstrap starting.
I20260812 02:36:19.896674 4749 tablet_bootstrap.cc:654] T ed5a63e0fdab476e95574f5860d8bf26 P bbea1665b24c4bf39ba99ca7696d1462: Neither blocks nor log segments found. Creating new log.
I20260812 02:36:19.897663 4749 log.cc:824] T ed5a63e0fdab476e95574f5860d8bf26 P bbea1665b24c4bf39ba99ca7696d1462: Log is configured to *not* fsync() on all Append() calls
I20260812 02:36:19.897835 4751 tablet_bootstrap.cc:654] T ed5a63e0fdab476e95574f5860d8bf26 P a9dc9ad78d1244baba20193904ca79ac: Neither blocks nor log segments found. Creating new log.
I20260812 02:36:19.897876 4750 tablet_bootstrap.cc:654] T ed5a63e0fdab476e95574f5860d8bf26 P e39c748b3f8e47b3842e7092fecee47f: Neither blocks nor log segments found. Creating new log.
I20260812 02:36:19.898797 4750 log.cc:824] T ed5a63e0fdab476e95574f5860d8bf26 P e39c748b3f8e47b3842e7092fecee47f: Log is configured to *not* fsync() on all Append() calls
I20260812 02:36:19.899047 4751 log.cc:824] T ed5a63e0fdab476e95574f5860d8bf26 P a9dc9ad78d1244baba20193904ca79ac: Log is configured to *not* fsync() on all Append() calls
I20260812 02:36:19.900131 4749 tablet_bootstrap.cc:492] T ed5a63e0fdab476e95574f5860d8bf26 P bbea1665b24c4bf39ba99ca7696d1462: No bootstrap required, opened a new log
I20260812 02:36:19.900352 4749 ts_tablet_manager.cc:1397] T ed5a63e0fdab476e95574f5860d8bf26 P bbea1665b24c4bf39ba99ca7696d1462: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260812 02:36:19.900911 4750 tablet_bootstrap.cc:492] T ed5a63e0fdab476e95574f5860d8bf26 P e39c748b3f8e47b3842e7092fecee47f: No bootstrap required, opened a new log
I20260812 02:36:19.901089 4750 ts_tablet_manager.cc:1397] T ed5a63e0fdab476e95574f5860d8bf26 P e39c748b3f8e47b3842e7092fecee47f: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260812 02:36:19.901189 4751 tablet_bootstrap.cc:492] T ed5a63e0fdab476e95574f5860d8bf26 P a9dc9ad78d1244baba20193904ca79ac: No bootstrap required, opened a new log
I20260812 02:36:19.901378 4751 ts_tablet_manager.cc:1397] T ed5a63e0fdab476e95574f5860d8bf26 P a9dc9ad78d1244baba20193904ca79ac: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260812 02:36:19.904597 4749 raft_consensus.cc:348] T ed5a63e0fdab476e95574f5860d8bf26 P bbea1665b24c4bf39ba99ca7696d1462 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9dc9ad78d1244baba20193904ca79ac" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 34831 } } peers { permanent_uuid: "e39c748b3f8e47b3842e7092fecee47f" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 37793 } } peers { permanent_uuid: "bbea1665b24c4bf39ba99ca7696d1462" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 33357 } }
I20260812 02:36:19.904807 4751 raft_consensus.cc:348] T ed5a63e0fdab476e95574f5860d8bf26 P a9dc9ad78d1244baba20193904ca79ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9dc9ad78d1244baba20193904ca79ac" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 34831 } } peers { permanent_uuid: "e39c748b3f8e47b3842e7092fecee47f" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 37793 } } peers { permanent_uuid: "bbea1665b24c4bf39ba99ca7696d1462" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 33357 } }
I20260812 02:36:19.905100 4751 raft_consensus.cc:374] T ed5a63e0fdab476e95574f5860d8bf26 P a9dc9ad78d1244baba20193904ca79ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:36:19.904986 4749 raft_consensus.cc:374] T ed5a63e0fdab476e95574f5860d8bf26 P bbea1665b24c4bf39ba99ca7696d1462 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:36:19.905175 4751 raft_consensus.cc:729] T ed5a63e0fdab476e95574f5860d8bf26 P a9dc9ad78d1244baba20193904ca79ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9dc9ad78d1244baba20193904ca79ac, State: Initialized, Role: FOLLOWER
I20260812 02:36:19.905172 4749 raft_consensus.cc:729] T ed5a63e0fdab476e95574f5860d8bf26 P bbea1665b24c4bf39ba99ca7696d1462 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbea1665b24c4bf39ba99ca7696d1462, State: Initialized, Role: FOLLOWER
I20260812 02:36:19.905443 4750 raft_consensus.cc:348] T ed5a63e0fdab476e95574f5860d8bf26 P e39c748b3f8e47b3842e7092fecee47f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9dc9ad78d1244baba20193904ca79ac" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 34831 } } peers { permanent_uuid: "e39c748b3f8e47b3842e7092fecee47f" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 37793 } } peers { permanent_uuid: "bbea1665b24c4bf39ba99ca7696d1462" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 33357 } }
I20260812 02:36:19.905694 4750 raft_consensus.cc:374] T ed5a63e0fdab476e95574f5860d8bf26 P e39c748b3f8e47b3842e7092fecee47f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:36:19.905620 4751 consensus_queue.cc:260] T ed5a63e0fdab476e95574f5860d8bf26 P a9dc9ad78d1244baba20193904ca79ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9dc9ad78d1244baba20193904ca79ac" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 34831 } } peers { permanent_uuid: "e39c748b3f8e47b3842e7092fecee47f" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 37793 } } peers { permanent_uuid: "bbea1665b24c4bf39ba99ca7696d1462" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 33357 } }
I20260812 02:36:19.905750 4750 raft_consensus.cc:729] T ed5a63e0fdab476e95574f5860d8bf26 P e39c748b3f8e47b3842e7092fecee47f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e39c748b3f8e47b3842e7092fecee47f, State: Initialized, Role: FOLLOWER
I20260812 02:36:19.905735 4749 consensus_queue.cc:260] T ed5a63e0fdab476e95574f5860d8bf26 P bbea1665b24c4bf39ba99ca7696d1462 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9dc9ad78d1244baba20193904ca79ac" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 34831 } } peers { permanent_uuid: "e39c748b3f8e47b3842e7092fecee47f" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 37793 } } peers { permanent_uuid: "bbea1665b24c4bf39ba99ca7696d1462" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 33357 } }
I20260812 02:36:19.906145 4750 consensus_queue.cc:260] T ed5a63e0fdab476e95574f5860d8bf26 P e39c748b3f8e47b3842e7092fecee47f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9dc9ad78d1244baba20193904ca79ac" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 34831 } } peers { permanent_uuid: "e39c748b3f8e47b3842e7092fecee47f" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 37793 } } peers { permanent_uuid: "bbea1665b24c4bf39ba99ca7696d1462" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 33357 } }
I20260812 02:36:19.906579 4578 heartbeater.cc:503] Master 127.14.103.62:42523 was elected leader, sending a full tablet report...
I20260812 02:36:19.906797 4749 ts_tablet_manager.cc:1428] T ed5a63e0fdab476e95574f5860d8bf26 P bbea1665b24c4bf39ba99ca7696d1462: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260812 02:36:19.906793 4751 ts_tablet_manager.cc:1428] T ed5a63e0fdab476e95574f5860d8bf26 P a9dc9ad78d1244baba20193904ca79ac: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260812 02:36:19.906988 4750 ts_tablet_manager.cc:1428] T ed5a63e0fdab476e95574f5860d8bf26 P e39c748b3f8e47b3842e7092fecee47f: Time spent starting tablet: real 0.006s user 0.002s sys 0.005s
I20260812 02:36:19.907159 4731 heartbeater.cc:503] Master 127.14.103.62:42523 was elected leader, sending a full tablet report...
I20260812 02:36:19.908169 4409 heartbeater.cc:503] Master 127.14.103.62:42523 was elected leader, sending a full tablet report...
I20260812 02:36:20.034890 4760 raft_consensus.cc:482] T ed5a63e0fdab476e95574f5860d8bf26 P bbea1665b24c4bf39ba99ca7696d1462 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 02:36:20.035208 4760 raft_consensus.cc:504] T ed5a63e0fdab476e95574f5860d8bf26 P bbea1665b24c4bf39ba99ca7696d1462 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9dc9ad78d1244baba20193904ca79ac" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 34831 } } peers { permanent_uuid: "e39c748b3f8e47b3842e7092fecee47f" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 37793 } } peers { permanent_uuid: "bbea1665b24c4bf39ba99ca7696d1462" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 33357 } }
I20260812 02:36:20.036594 4760 leader_election.cc:288] T ed5a63e0fdab476e95574f5860d8bf26 P bbea1665b24c4bf39ba99ca7696d1462 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a9dc9ad78d1244baba20193904ca79ac (127.14.103.3:34831), e39c748b3f8e47b3842e7092fecee47f (127.14.103.1:37793)
I20260812 02:36:20.040908 4748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:20.037409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.2:59859 (local address 127.14.103.1:37793)
0812 02:36:20.037542 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0812 02:36:20.037546 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:36:20.037624 (+ 78us) server_negotiation.cc:407] Connection header received
0812 02:36:20.037723 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:36:20.037726 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:36:20.037771 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:36:20.037846 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:36:20.038105 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:20.038671 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:20.039920 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:20.040353 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:20.040481 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:36:20.040536 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0812 02:36:20.040700 (+ 164us) server_negotiation.cc:299] Negotiation successful
0812 02:36:20.040786 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260812 02:36:20.040904 4762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:20.037249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.1:37793 (local address 127.14.103.2:59859)
0812 02:36:20.037552 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0812 02:36:20.037569 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 02:36:20.037652 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:36:20.037874 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:36:20.037876 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:36:20.037889 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:36:20.037998 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:36:20.038003 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:36:20.038784 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:36:20.038790 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:36:20.039794 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:36:20.039804 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:36:20.040471 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:36:20.040475 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:36:20.040574 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:36:20.040597 (+ 23us) client_negotiation.cc:769] Sending connection context
0812 02:36:20.040665 (+ 68us) client_negotiation.cc:240] Negotiation successful
0812 02:36:20.040738 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"mutex_wait_us":75,"thread_start_us":135,"threads_started":1}
I20260812 02:36:20.040925 4581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:20.036927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.3:34831 (local address 127.14.103.2:48167)
0812 02:36:20.037058 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0812 02:36:20.037077 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 02:36:20.037185 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:36:20.037933 (+ 748us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:36:20.037936 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:36:20.037954 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:36:20.038095 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:36:20.038102 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:36:20.038849 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:36:20.038852 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:36:20.039798 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:36:20.039807 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:36:20.040511 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:36:20.040515 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:36:20.040629 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:36:20.040664 (+ 35us) client_negotiation.cc:769] Sending connection context
0812 02:36:20.040747 (+ 83us) client_negotiation.cc:240] Negotiation successful
0812 02:36:20.040819 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47,"mutex_wait_us":93}
I20260812 02:36:20.040969 4743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:20.037003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.2:48167 (local address 127.14.103.3:34831)
0812 02:36:20.037133 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0812 02:36:20.037137 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:36:20.037159 (+ 22us) server_negotiation.cc:407] Connection header received
0812 02:36:20.037351 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:36:20.037354 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:36:20.037407 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:36:20.037526 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:36:20.038222 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:20.038703 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:20.039927 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:20.040363 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:20.040496 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:36:20.040549 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0812 02:36:20.040781 (+ 232us) server_negotiation.cc:299] Negotiation successful
0812 02:36:20.040851 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260812 02:36:20.041659 4681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed5a63e0fdab476e95574f5860d8bf26" candidate_uuid: "bbea1665b24c4bf39ba99ca7696d1462" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9dc9ad78d1244baba20193904ca79ac" is_pre_election: true
I20260812 02:36:20.041983 4681 raft_consensus.cc:2454] T ed5a63e0fdab476e95574f5860d8bf26 P a9dc9ad78d1244baba20193904ca79ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbea1665b24c4bf39ba99ca7696d1462 in term 0.
I20260812 02:36:20.042544 4436 leader_election.cc:302] T ed5a63e0fdab476e95574f5860d8bf26 P bbea1665b24c4bf39ba99ca7696d1462 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a9dc9ad78d1244baba20193904ca79ac, bbea1665b24c4bf39ba99ca7696d1462; no voters:
I20260812 02:36:20.042564 4352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed5a63e0fdab476e95574f5860d8bf26" candidate_uuid: "bbea1665b24c4bf39ba99ca7696d1462" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e39c748b3f8e47b3842e7092fecee47f" is_pre_election: true
I20260812 02:36:20.042816 4352 raft_consensus.cc:2454] T ed5a63e0fdab476e95574f5860d8bf26 P e39c748b3f8e47b3842e7092fecee47f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbea1665b24c4bf39ba99ca7696d1462 in term 0.
I20260812 02:36:20.042876 4760 raft_consensus.cc:2781] T ed5a63e0fdab476e95574f5860d8bf26 P bbea1665b24c4bf39ba99ca7696d1462 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 02:36:20.042950 4760 raft_consensus.cc:482] T ed5a63e0fdab476e95574f5860d8bf26 P bbea1665b24c4bf39ba99ca7696d1462 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 02:36:20.043035 4760 raft_consensus.cc:3037] T ed5a63e0fdab476e95574f5860d8bf26 P bbea1665b24c4bf39ba99ca7696d1462 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:36:20.044123 4760 raft_consensus.cc:504] T ed5a63e0fdab476e95574f5860d8bf26 P bbea1665b24c4bf39ba99ca7696d1462 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9dc9ad78d1244baba20193904ca79ac" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 34831 } } peers { permanent_uuid: "e39c748b3f8e47b3842e7092fecee47f" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 37793 } } peers { permanent_uuid: "bbea1665b24c4bf39ba99ca7696d1462" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 33357 } }
I20260812 02:36:20.044510 4760 leader_election.cc:288] T ed5a63e0fdab476e95574f5860d8bf26 P bbea1665b24c4bf39ba99ca7696d1462 [CANDIDATE]: Term 1 election: Requested vote from peers a9dc9ad78d1244baba20193904ca79ac (127.14.103.3:34831), e39c748b3f8e47b3842e7092fecee47f (127.14.103.1:37793)
I20260812 02:36:20.044919 4352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed5a63e0fdab476e95574f5860d8bf26" candidate_uuid: "bbea1665b24c4bf39ba99ca7696d1462" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e39c748b3f8e47b3842e7092fecee47f"
I20260812 02:36:20.044936 4681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed5a63e0fdab476e95574f5860d8bf26" candidate_uuid: "bbea1665b24c4bf39ba99ca7696d1462" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9dc9ad78d1244baba20193904ca79ac"
I20260812 02:36:20.045053 4681 raft_consensus.cc:3037] T ed5a63e0fdab476e95574f5860d8bf26 P a9dc9ad78d1244baba20193904ca79ac [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:36:20.045049 4352 raft_consensus.cc:3037] T ed5a63e0fdab476e95574f5860d8bf26 P e39c748b3f8e47b3842e7092fecee47f [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:36:20.046365 4352 raft_consensus.cc:2454] T ed5a63e0fdab476e95574f5860d8bf26 P e39c748b3f8e47b3842e7092fecee47f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbea1665b24c4bf39ba99ca7696d1462 in term 1.
I20260812 02:36:20.046365 4681 raft_consensus.cc:2454] T ed5a63e0fdab476e95574f5860d8bf26 P a9dc9ad78d1244baba20193904ca79ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbea1665b24c4bf39ba99ca7696d1462 in term 1.
I20260812 02:36:20.046756 4436 leader_election.cc:302] T ed5a63e0fdab476e95574f5860d8bf26 P bbea1665b24c4bf39ba99ca7696d1462 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a9dc9ad78d1244baba20193904ca79ac, bbea1665b24c4bf39ba99ca7696d1462; no voters:
I20260812 02:36:20.047048 4760 raft_consensus.cc:2781] T ed5a63e0fdab476e95574f5860d8bf26 P bbea1665b24c4bf39ba99ca7696d1462 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:36:20.047309 4760 raft_consensus.cc:686] T ed5a63e0fdab476e95574f5860d8bf26 P bbea1665b24c4bf39ba99ca7696d1462 [term 1 LEADER]: Becoming Leader. State: Replica: bbea1665b24c4bf39ba99ca7696d1462, State: Running, Role: LEADER
I20260812 02:36:20.047665 4760 consensus_queue.cc:237] T ed5a63e0fdab476e95574f5860d8bf26 P bbea1665b24c4bf39ba99ca7696d1462 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9dc9ad78d1244baba20193904ca79ac" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 34831 } } peers { permanent_uuid: "e39c748b3f8e47b3842e7092fecee47f" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 37793 } } peers { permanent_uuid: "bbea1665b24c4bf39ba99ca7696d1462" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 33357 } }
I20260812 02:36:20.051128 4185 catalog_manager.cc:5495] T ed5a63e0fdab476e95574f5860d8bf26 P bbea1665b24c4bf39ba99ca7696d1462 reported cstate change: term changed from 0 to 1, leader changed from <none> to bbea1665b24c4bf39ba99ca7696d1462 (127.14.103.2). New cstate: current_term: 1 leader_uuid: "bbea1665b24c4bf39ba99ca7696d1462" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9dc9ad78d1244baba20193904ca79ac" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 34831 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e39c748b3f8e47b3842e7092fecee47f" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 37793 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bbea1665b24c4bf39ba99ca7696d1462" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 33357 } health_report { overall_health: HEALTHY } } }
I20260812 02:36:20.073257 4747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:20.069186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34260 (local address 127.14.103.2:33357)
0812 02:36:20.069331 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0812 02:36:20.069336 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:36:20.069355 (+ 19us) server_negotiation.cc:407] Connection header received
0812 02:36:20.069435 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:36:20.069438 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:36:20.069494 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:36:20.069600 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 02:36:20.069948 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:20.070570 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:20.071559 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:20.071791 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:20.071967 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:36:20.072271 (+ 304us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 02:36:20.072776 (+ 505us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 02:36:20.072893 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0812 02:36:20.072953 (+ 60us) server_negotiation.cc:299] Negotiation successful
0812 02:36:20.073032 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260812 02:36:20.079808 4681 raft_consensus.cc:1261] T ed5a63e0fdab476e95574f5860d8bf26 P a9dc9ad78d1244baba20193904ca79ac [term 1 FOLLOWER]: Refusing update from remote peer bbea1665b24c4bf39ba99ca7696d1462: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:36:20.079823 4352 raft_consensus.cc:1261] T ed5a63e0fdab476e95574f5860d8bf26 P e39c748b3f8e47b3842e7092fecee47f [term 1 FOLLOWER]: Refusing update from remote peer bbea1665b24c4bf39ba99ca7696d1462: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260812 02:36:20.080374 4732 tablet.cc:2369] T ed5a63e0fdab476e95574f5860d8bf26 P a9dc9ad78d1244baba20193904ca79ac: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 02:36:20.080637 4760 consensus_queue.cc:1035] T ed5a63e0fdab476e95574f5860d8bf26 P bbea1665b24c4bf39ba99ca7696d1462 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9dc9ad78d1244baba20193904ca79ac" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 34831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:36:20.081018 4763 consensus_queue.cc:1035] T ed5a63e0fdab476e95574f5860d8bf26 P bbea1665b24c4bf39ba99ca7696d1462 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e39c748b3f8e47b3842e7092fecee47f" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 37793 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:36:20.092999 4769 mvcc.cc:205] Tried to move back new op lower bound from 7317512929598746624 to 7317512929477705728. Current Snapshot: MvccSnapshot[applied={T|T < 7317512929598746624}]
I20260812 02:36:20.094105 4768 mvcc.cc:205] Tried to move back new op lower bound from 7317512929598746624 to 7317512929477705728. Current Snapshot: MvccSnapshot[applied={T|T < 7317512929598746624}]
I20260812 02:36:20.099687 4178 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58412:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 02:36:20.099880 4178 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58412:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 02:36:20.102277 4178 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cfca43c27bef497f88696682fae47b1f: Sending DeleteTablet for 3 replicas of tablet ed5a63e0fdab476e95574f5860d8bf26
I20260812 02:36:20.103214 4328 tablet_service.cc:1514] Processing DeleteTablet for tablet ed5a63e0fdab476e95574f5860d8bf26 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:36:20 UTC) from {username='kudu', principal='kudu/127.14.103.62@KRBTEST.COM'} at 127.0.0.1:53076
I20260812 02:36:20.103261 4492 tablet_service.cc:1514] Processing DeleteTablet for tablet ed5a63e0fdab476e95574f5860d8bf26 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:36:20 UTC) from {username='kudu', principal='kudu/127.14.103.62@KRBTEST.COM'} at 127.0.0.1:34254
I20260812 02:36:20.103457 4659 tablet_service.cc:1514] Processing DeleteTablet for tablet ed5a63e0fdab476e95574f5860d8bf26 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:36:20 UTC) from {username='kudu', principal='kudu/127.14.103.62@KRBTEST.COM'} at 127.0.0.1:35628
I20260812 02:36:20.103746 4778 tablet_replica.cc:330] T ed5a63e0fdab476e95574f5860d8bf26 P bbea1665b24c4bf39ba99ca7696d1462: stopping tablet replica
I20260812 02:36:20.104106 4778 raft_consensus.cc:2229] T ed5a63e0fdab476e95574f5860d8bf26 P bbea1665b24c4bf39ba99ca7696d1462 [term 1 LEADER]: Raft consensus shutting down.
I20260812 02:36:20.104133 4780 tablet_replica.cc:330] T ed5a63e0fdab476e95574f5860d8bf26 P a9dc9ad78d1244baba20193904ca79ac: stopping tablet replica
I20260812 02:36:20.104233 14748 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbep4M4/build/debug/bin/kudu with pid 4257
I20260812 02:36:20.104794 4780 raft_consensus.cc:2229] T ed5a63e0fdab476e95574f5860d8bf26 P a9dc9ad78d1244baba20193904ca79ac [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 02:36:20.105082 4780 raft_consensus.cc:2258] T ed5a63e0fdab476e95574f5860d8bf26 P a9dc9ad78d1244baba20193904ca79ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:36:20.107223 4778 raft_consensus.cc:2258] T ed5a63e0fdab476e95574f5860d8bf26 P bbea1665b24c4bf39ba99ca7696d1462 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:36:20.108351 4780 ts_tablet_manager.cc:1905] T ed5a63e0fdab476e95574f5860d8bf26 P a9dc9ad78d1244baba20193904ca79ac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:36:20.108672 4778 ts_tablet_manager.cc:1905] T ed5a63e0fdab476e95574f5860d8bf26 P bbea1665b24c4bf39ba99ca7696d1462: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:36:20.111394 4780 ts_tablet_manager.cc:1918] T ed5a63e0fdab476e95574f5860d8bf26 P a9dc9ad78d1244baba20193904ca79ac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 02:36:20.111513 4780 log.cc:1196] T ed5a63e0fdab476e95574f5860d8bf26 P a9dc9ad78d1244baba20193904ca79ac: Deleting WAL directory at /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501898713699-14748-0/minicluster-data/ts-2/wal/wals/ed5a63e0fdab476e95574f5860d8bf26
I20260812 02:36:20.111884 4780 ts_tablet_manager.cc:1939] T ed5a63e0fdab476e95574f5860d8bf26 P a9dc9ad78d1244baba20193904ca79ac: Deleting consensus metadata
I20260812 02:36:20.111999 4778 ts_tablet_manager.cc:1918] T ed5a63e0fdab476e95574f5860d8bf26 P bbea1665b24c4bf39ba99ca7696d1462: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 02:36:20.112119 4778 log.cc:1196] T ed5a63e0fdab476e95574f5860d8bf26 P bbea1665b24c4bf39ba99ca7696d1462: Deleting WAL directory at /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786501898713699-14748-0/minicluster-data/ts-1/wal/wals/ed5a63e0fdab476e95574f5860d8bf26
I20260812 02:36:20.112468 4778 ts_tablet_manager.cc:1939] T ed5a63e0fdab476e95574f5860d8bf26 P bbea1665b24c4bf39ba99ca7696d1462: Deleting consensus metadata
I20260812 02:36:20.113250 4167 catalog_manager.cc:4841] TS a9dc9ad78d1244baba20193904ca79ac (127.14.103.3:34831): tablet ed5a63e0fdab476e95574f5860d8bf26 (table test-table [id=29f5f55bd7584e6787d8bc5615102620]) successfully deleted
I20260812 02:36:20.113626 4168 catalog_manager.cc:4841] TS bbea1665b24c4bf39ba99ca7696d1462 (127.14.103.2:33357): tablet ed5a63e0fdab476e95574f5860d8bf26 (table test-table [id=29f5f55bd7584e6787d8bc5615102620]) successfully deleted
W20260812 02:36:20.117049 4167 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.103.1:37793 (error 108)
I20260812 02:36:20.117172 14748 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbep4M4/build/debug/bin/kudu with pid 4414
I20260812 02:36:20.121508 4745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:20.121034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.1:37793 (local address 127.0.0.1:53080)
0812 02:36:20.121326 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0812 02:36:20.121415 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.1:37793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
W20260812 02:36:20.121830 4167 catalog_manager.cc:4584] TS e39c748b3f8e47b3842e7092fecee47f (127.14.103.1:37793): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ed5a63e0fdab476e95574f5860d8bf26: Network error: Client connection negotiation failed: client connection to 127.14.103.1:37793: connect: Connection refused (error 111)
I20260812 02:36:20.127282 14748 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbep4M4/build/debug/bin/kudu with pid 4584
I20260812 02:36:20.136013 14748 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbep4M4/build/debug/bin/kudu with pid 4149
2026-08-12T02:36:20Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2600 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-08-12T02:36:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T02:36:20Z Disabled control of system clock
I20260812 02:36:20.173933 14748 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbep4M4/build/debug/bin/kudu
/tmp/dist-test-taskbep4M4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786501898713699-14748-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786501898713699-14748-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786501898713699-14748-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786501898713699-14748-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.103.62:46141
--webserver_interface=127.14.103.62
--webserver_port=0
--builtin_ntp_servers=127.14.103.20:33535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.103.62:46141
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260812 02:36:20.294288 4786 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:36:20.294606 4786 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:36:20.294646 4786 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:36:20.298790 4786 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:36:20.298919 4786 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:36:20.298938 4786 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:36:20.298977 4786 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:36:20.298993 4786 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 02:36:20.304114 4786 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.103.20:33535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786501898713699-14748-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786501898713699-14748-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.103.62:46141
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.103.62:46141
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786501898713699-14748-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.103.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4786
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786501898713699-14748-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:36:20.305244 4786 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:36:20.306602 4786 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:36:20.315809 4791 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:36:20.315809 4795 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:36:20.315855 4786 server_base.cc:1034] running on GCE node
W20260812 02:36:20.315809 4792 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:36:20.316660 4786 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:36:20.317912 4786 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:36:20.319060 4786 hybrid_clock.cc:648] HybridClock initialized: now 1786502180319038 us; error 33 us; skew 500 ppm
I20260812 02:36:20.321375 4786 webserver.cc:460] Webserver started at http://127.14.103.62:35263/ using document root <none> and password file <none>
I20260812 02:36:20.322060 4786 fs_manager.cc:359] Metadata directory not provided
I20260812 02:36:20.322137 4786 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:36:20.322409 4786 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:36:20.324966 4786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786501898713699-14748-0/minicluster-data/master-0/data/instance:
uuid: "12f7cf80816147a894b04155e4ab7032"
format_stamp: "Formatted at 2026-08-12 02:36:20 on dist-test-slave-2w3w"
server_key: "8f23bfd869a0157c0b1d7259a5b9742c"
server_key_iv: "164f786008f2c94e905ede736ed58db1"
server_key_version: "encryptionkey@0"
I20260812 02:36:20.325598 4786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786501898713699-14748-0/minicluster-data/master-0/wal/instance:
uuid: "12f7cf80816147a894b04155e4ab7032"
format_stamp: "Formatted at 2026-08-12 02:36:20 on dist-test-slave-2w3w"
server_key: "8f23bfd869a0157c0b1d7259a5b9742c"
server_key_iv: "164f786008f2c94e905ede736ed58db1"
server_key_version: "encryptionkey@0"
I20260812 02:36:20.329689 4786 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260812 02:36:20.332456 4801 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:36:20.334049 4786 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260812 02:36:20.334221 4786 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786501898713699-14748-0/minicluster-data/master-0/data,/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786501898713699-14748-0/minicluster-data/master-0/wal
uuid: "12f7cf80816147a894b04155e4ab7032"
format_stamp: "Formatted at 2026-08-12 02:36:20 on dist-test-slave-2w3w"
server_key: "8f23bfd869a0157c0b1d7259a5b9742c"
server_key_iv: "164f786008f2c94e905ede736ed58db1"
server_key_version: "encryptionkey@0"
I20260812 02:36:20.334353 4786 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786501898713699-14748-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786501898713699-14748-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786501898713699-14748-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:36:20.363294 4786 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:36:20.364058 4786 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:36:20.364276 4786 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:36:20.371678 4786 rpc_server.cc:307] RPC server started. Bound to: 127.14.103.62:46141
I20260812 02:36:20.371699 4867 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.103.62:46141 every 8 connection(s)
I20260812 02:36:20.372757 4786 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786501898713699-14748-0/minicluster-data/master-0/data/info.pb
I20260812 02:36:20.375989 4868 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:36:20.381781 14748 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbep4M4/build/debug/bin/kudu as pid 4786
I20260812 02:36:20.381832 4868 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12f7cf80816147a894b04155e4ab7032: Bootstrap starting.
I20260812 02:36:20.381968 14748 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786501898713699-14748-0/minicluster-data/master-0/wal/instance
I20260812 02:36:20.382252 14748 external_mini_cluster.cc:1428] Setting key a50995f2438a3f56213758738f935e06
I20260812 02:36:20.385334 4868 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 12f7cf80816147a894b04155e4ab7032: Neither blocks nor log segments found. Creating new log.
I20260812 02:36:20.386317 4868 log.cc:824] T 00000000000000000000000000000000 P 12f7cf80816147a894b04155e4ab7032: Log is configured to *not* fsync() on all Append() calls
I20260812 02:36:20.388614 4868 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12f7cf80816147a894b04155e4ab7032: No bootstrap required, opened a new log
I20260812 02:36:20.392364 4871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:20.383686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41296 (local address 127.14.103.62:46141)
0812 02:36:20.384318 (+ 632us) server_negotiation.cc:206] Beginning negotiation
0812 02:36:20.384334 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0812 02:36:20.384371 (+ 37us) server_negotiation.cc:407] Connection header received
0812 02:36:20.385051 (+ 680us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:36:20.385068 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:36:20.385526 (+ 458us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:36:20.385810 (+ 284us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:36:20.386176 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:20.387295 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:20.388476 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:20.389115 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:20.389270 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:36:20.390285 (+ 1015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:36:20.390290 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:36:20.390303 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:36:20.390831 (+ 528us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:36:20.390903 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:36:20.390925 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:36:20.391096 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0812 02:36:20.391277 (+ 181us) server_negotiation.cc:299] Negotiation successful
0812 02:36:20.391590 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":122,"threads_started":1}
I20260812 02:36:20.393100 4868 raft_consensus.cc:348] T 00000000000000000000000000000000 P 12f7cf80816147a894b04155e4ab7032 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12f7cf80816147a894b04155e4ab7032" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 46141 } }
I20260812 02:36:20.393366 4868 raft_consensus.cc:374] T 00000000000000000000000000000000 P 12f7cf80816147a894b04155e4ab7032 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:36:20.393445 4868 raft_consensus.cc:729] T 00000000000000000000000000000000 P 12f7cf80816147a894b04155e4ab7032 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12f7cf80816147a894b04155e4ab7032, State: Initialized, Role: FOLLOWER
I20260812 02:36:20.394143 4868 consensus_queue.cc:260] T 00000000000000000000000000000000 P 12f7cf80816147a894b04155e4ab7032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12f7cf80816147a894b04155e4ab7032" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 46141 } }
I20260812 02:36:20.394337 4868 raft_consensus.cc:388] T 00000000000000000000000000000000 P 12f7cf80816147a894b04155e4ab7032 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 02:36:20.394433 4868 raft_consensus.cc:482] T 00000000000000000000000000000000 P 12f7cf80816147a894b04155e4ab7032 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 02:36:20.394531 4868 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 12f7cf80816147a894b04155e4ab7032 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:36:20.395709 4868 raft_consensus.cc:504] T 00000000000000000000000000000000 P 12f7cf80816147a894b04155e4ab7032 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12f7cf80816147a894b04155e4ab7032" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 46141 } }
I20260812 02:36:20.396011 4868 leader_election.cc:302] T 00000000000000000000000000000000 P 12f7cf80816147a894b04155e4ab7032 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 12f7cf80816147a894b04155e4ab7032; no voters:
I20260812 02:36:20.396304 4868 leader_election.cc:288] T 00000000000000000000000000000000 P 12f7cf80816147a894b04155e4ab7032 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 02:36:20.396442 4873 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 12f7cf80816147a894b04155e4ab7032 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:36:20.396698 4873 raft_consensus.cc:686] T 00000000000000000000000000000000 P 12f7cf80816147a894b04155e4ab7032 [term 1 LEADER]: Becoming Leader. State: Replica: 12f7cf80816147a894b04155e4ab7032, State: Running, Role: LEADER
I20260812 02:36:20.397032 4873 consensus_queue.cc:237] T 00000000000000000000000000000000 P 12f7cf80816147a894b04155e4ab7032 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12f7cf80816147a894b04155e4ab7032" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 46141 } }
I20260812 02:36:20.397576 4868 sys_catalog.cc:564] T 00000000000000000000000000000000 P 12f7cf80816147a894b04155e4ab7032 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:36:20.399308 4875 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12f7cf80816147a894b04155e4ab7032 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 12f7cf80816147a894b04155e4ab7032. Latest consensus state: current_term: 1 leader_uuid: "12f7cf80816147a894b04155e4ab7032" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12f7cf80816147a894b04155e4ab7032" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 46141 } } }
I20260812 02:36:20.399475 4875 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12f7cf80816147a894b04155e4ab7032 [sys.catalog]: This master's current role is: LEADER
I20260812 02:36:20.399704 4874 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12f7cf80816147a894b04155e4ab7032 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "12f7cf80816147a894b04155e4ab7032" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12f7cf80816147a894b04155e4ab7032" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 46141 } } }
I20260812 02:36:20.399796 4874 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12f7cf80816147a894b04155e4ab7032 [sys.catalog]: This master's current role is: LEADER
W20260812 02:36:20.403188 4890 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 12f7cf80816147a894b04155e4ab7032: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 02:36:20.403278 4890 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 02:36:20.403366 4888 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 02:36:20.404064 4888 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 02:36:20.409758 4888 catalog_manager.cc:1347] Generated new cluster ID: 21c231dd290b456ea7aa8fd311f2fe53
I20260812 02:36:20.409893 4888 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 02:36:20.426574 4888 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 02:36:20.427963 4888 catalog_manager.cc:1504] Loading token signing keys...
I20260812 02:36:20.439021 4888 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 12f7cf80816147a894b04155e4ab7032: Generated new TSK 0
I20260812 02:36:20.440008 4888 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 02:36:20.443460 14748 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbep4M4/build/debug/bin/kudu
/tmp/dist-test-taskbep4M4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786501898713699-14748-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786501898713699-14748-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786501898713699-14748-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786501898713699-14748-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.103.1:0
--local_ip_for_outbound_sockets=127.14.103.1
--webserver_interface=127.14.103.1
--webserver_port=0
--tserver_master_addrs=127.14.103.62:46141
--builtin_ntp_servers=127.14.103.20:33535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260812 02:36:20.560961 4894 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:36:20.561303 4894 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:36:20.561386 4894 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:36:20.565714 4894 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:36:20.565824 4894 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:36:20.565923 4894 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.103.1
I20260812 02:36:20.570885 4894 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.103.20:33535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786501898713699-14748-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786501898713699-14748-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.103.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786501898713699-14748-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.103.1
--webserver_port=0
--tserver_master_addrs=127.14.103.62:46141
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4894
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.103.1
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786501898713699-14748-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:36:20.572119 4894 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:36:20.573542 4894 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:36:20.584190 4902 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:36:20.584192 4900 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:36:20.584192 4899 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:36:20.584518 4894 server_base.cc:1034] running on GCE node
I20260812 02:36:20.584988 4894 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:36:20.585705 4894 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:36:20.586897 4894 hybrid_clock.cc:648] HybridClock initialized: now 1786502180586880 us; error 31 us; skew 500 ppm
I20260812 02:36:20.593422 4894 webserver.cc:460] Webserver started at http://127.14.103.1:37923/ using document root <none> and password file <none>
I20260812 02:36:20.594022 4894 fs_manager.cc:359] Metadata directory not provided
I20260812 02:36:20.594091 4894 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:36:20.594300 4894 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:36:20.596536 4894 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786501898713699-14748-0/minicluster-data/ts-0/data/instance:
uuid: "fb1cbd7f4a1841dc8a34b2bdcc615e14"
format_stamp: "Formatted at 2026-08-12 02:36:20 on dist-test-slave-2w3w"
server_key: "9d7530c6e4f1a2e025ae891cfd32fea3"
server_key_iv: "939adcf28b9e2f230083f54d28a9e812"
server_key_version: "encryptionkey@0"
I20260812 02:36:20.597082 4894 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786501898713699-14748-0/minicluster-data/ts-0/wal/instance:
uuid: "fb1cbd7f4a1841dc8a34b2bdcc615e14"
format_stamp: "Formatted at 2026-08-12 02:36:20 on dist-test-slave-2w3w"
server_key: "9d7530c6e4f1a2e025ae891cfd32fea3"
server_key_iv: "939adcf28b9e2f230083f54d28a9e812"
server_key_version: "encryptionkey@0"
I20260812 02:36:20.601034 4894 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260812 02:36:20.603575 4909 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:36:20.604861 4894 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260812 02:36:20.605008 4894 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786501898713699-14748-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786501898713699-14748-0/minicluster-data/ts-0/wal
uuid: "fb1cbd7f4a1841dc8a34b2bdcc615e14"
format_stamp: "Formatted at 2026-08-12 02:36:20 on dist-test-slave-2w3w"
server_key: "9d7530c6e4f1a2e025ae891cfd32fea3"
server_key_iv: "939adcf28b9e2f230083f54d28a9e812"
server_key_version: "encryptionkey@0"
I20260812 02:36:20.605131 4894 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786501898713699-14748-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786501898713699-14748-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786501898713699-14748-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:36:20.638250 4894 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:36:20.639065 4894 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:36:20.639271 4894 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:36:20.639919 4894 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:36:20.640892 4894 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:36:20.640962 4894 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:36:20.641034 4894 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:36:20.641074 4894 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:36:20.650027 4894 rpc_server.cc:307] RPC server started. Bound to: 127.14.103.1:41463
I20260812 02:36:20.650051 5045 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.103.1:41463 every 8 connection(s)
I20260812 02:36:20.650950 4894 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786501898713699-14748-0/minicluster-data/ts-0/data/info.pb
I20260812 02:36:20.651939 14748 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbep4M4/build/debug/bin/kudu as pid 4894
I20260812 02:36:20.652091 14748 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786501898713699-14748-0/minicluster-data/ts-0/wal/instance
I20260812 02:36:20.652390 14748 external_mini_cluster.cc:1428] Setting key b75f1aeccedb88ca0f84a336d718d489
I20260812 02:36:20.654542 14748 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbep4M4/build/debug/bin/kudu
/tmp/dist-test-taskbep4M4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786501898713699-14748-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786501898713699-14748-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786501898713699-14748-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786501898713699-14748-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.103.2:0
--local_ip_for_outbound_sockets=127.14.103.2
--webserver_interface=127.14.103.2
--webserver_port=0
--tserver_master_addrs=127.14.103.62:46141
--builtin_ntp_servers=127.14.103.20:33535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260812 02:36:20.662120 4871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:20.653935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.1:35837 (local address 127.14.103.62:46141)
0812 02:36:20.654061 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0812 02:36:20.654064 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 02:36:20.655419 (+ 1355us) server_negotiation.cc:407] Connection header received
0812 02:36:20.657114 (+ 1695us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:36:20.657117 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:36:20.657163 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:36:20.657241 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:36:20.658307 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:20.658862 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:20.660108 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:20.660336 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:20.660448 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:36:20.661606 (+ 1158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:36:20.661609 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:36:20.661612 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:36:20.661624 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:36:20.661643 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:36:20.661650 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:36:20.661724 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0812 02:36:20.661880 (+ 156us) server_negotiation.cc:299] Negotiation successful
0812 02:36:20.661947 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260812 02:36:20.663193 5049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:20.654495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:46141 (local address 127.14.103.1:35837)
0812 02:36:20.655276 (+ 781us) negotiation.cc:107] Waiting for socket to connect
0812 02:36:20.655318 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0812 02:36:20.656852 (+ 1534us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:36:20.657454 (+ 602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:36:20.657464 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:36:20.657584 (+ 120us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:36:20.658162 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:36:20.658176 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:36:20.659022 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:36:20.659025 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:36:20.659979 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:36:20.659989 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:36:20.660491 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:36:20.660495 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:36:20.660662 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:36:20.661342 (+ 680us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 02:36:20.661344 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:36:20.661386 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 02:36:20.661449 (+ 63us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:36:20.661739 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:36:20.661754 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:36:20.661759 (+ 5us) client_negotiation.cc:769] Sending connection context
0812 02:36:20.661912 (+ 153us) client_negotiation.cc:240] Negotiation successful
0812 02:36:20.662199 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":169,"threads_started":1}
I20260812 02:36:20.664521 5046 heartbeater.cc:347] Connected to a master server at 127.14.103.62:46141
I20260812 02:36:20.664763 5046 heartbeater.cc:464] Registering TS with master...
I20260812 02:36:20.665280 5046 heartbeater.cc:511] Master 127.14.103.62:46141 requested a full tablet report, sending...
I20260812 02:36:20.666931 4821 ts_manager.cc:194] Registered new tserver with Master: fb1cbd7f4a1841dc8a34b2bdcc615e14 (127.14.103.1:41463)
I20260812 02:36:20.668537 4821 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.103.1:35837
W20260812 02:36:20.774238 5050 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:36:20.774531 5050 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:36:20.774571 5050 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:36:20.778703 5050 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:36:20.778779 5050 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:36:20.778836 5050 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.103.2
I20260812 02:36:20.783931 5050 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.103.20:33535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786501898713699-14748-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786501898713699-14748-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.103.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786501898713699-14748-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.103.2
--webserver_port=0
--tserver_master_addrs=127.14.103.62:46141
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5050
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.103.2
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786501898713699-14748-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:36:20.784999 5050 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:36:20.786334 5050 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:36:20.796451 5057 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:36:20.796455 5060 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:36:20.796641 5050 server_base.cc:1034] running on GCE node
W20260812 02:36:20.796455 5058 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:36:20.797258 5050 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:36:20.797930 5050 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:36:20.799106 5050 hybrid_clock.cc:648] HybridClock initialized: now 1786502180799066 us; error 96 us; skew 500 ppm
I20260812 02:36:20.801024 5050 webserver.cc:460] Webserver started at http://127.14.103.2:37609/ using document root <none> and password file <none>
I20260812 02:36:20.801617 5050 fs_manager.cc:359] Metadata directory not provided
I20260812 02:36:20.801728 5050 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:36:20.801942 5050 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:36:20.804162 5050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786501898713699-14748-0/minicluster-data/ts-1/data/instance:
uuid: "ac824397231e44d3ac7ac89e7b8e83a5"
format_stamp: "Formatted at 2026-08-12 02:36:20 on dist-test-slave-2w3w"
server_key: "91b6439c76167f98075c74b4a6579955"
server_key_iv: "66061ee9fa0e9f861fe1635b6025967c"
server_key_version: "encryptionkey@0"
I20260812 02:36:20.804755 5050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786501898713699-14748-0/minicluster-data/ts-1/wal/instance:
uuid: "ac824397231e44d3ac7ac89e7b8e83a5"
format_stamp: "Formatted at 2026-08-12 02:36:20 on dist-test-slave-2w3w"
server_key: "91b6439c76167f98075c74b4a6579955"
server_key_iv: "66061ee9fa0e9f861fe1635b6025967c"
server_key_version: "encryptionkey@0"
I20260812 02:36:20.808636 5050 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260812 02:36:20.811252 5067 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:36:20.812490 5050 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260812 02:36:20.812620 5050 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786501898713699-14748-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786501898713699-14748-0/minicluster-data/ts-1/wal
uuid: "ac824397231e44d3ac7ac89e7b8e83a5"
format_stamp: "Formatted at 2026-08-12 02:36:20 on dist-test-slave-2w3w"
server_key: "91b6439c76167f98075c74b4a6579955"
server_key_iv: "66061ee9fa0e9f861fe1635b6025967c"
server_key_version: "encryptionkey@0"
I20260812 02:36:20.812736 5050 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786501898713699-14748-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786501898713699-14748-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786501898713699-14748-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:36:20.826062 5050 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:36:20.826818 5050 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:36:20.827085 5050 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:36:20.827710 5050 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:36:20.828691 5050 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:36:20.828769 5050 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:36:20.828871 5050 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:36:20.828918 5050 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:36:20.837970 5050 rpc_server.cc:307] RPC server started. Bound to: 127.14.103.2:41515
I20260812 02:36:20.838027 5195 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.103.2:41515 every 8 connection(s)
I20260812 02:36:20.839156 5050 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786501898713699-14748-0/minicluster-data/ts-1/data/info.pb
I20260812 02:36:20.842633 14748 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbep4M4/build/debug/bin/kudu as pid 5050
I20260812 02:36:20.842803 14748 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786501898713699-14748-0/minicluster-data/ts-1/wal/instance
I20260812 02:36:20.843173 14748 external_mini_cluster.cc:1428] Setting key bb9c69b65c3c55b22d765e9e8c7db37f
I20260812 02:36:20.845508 14748 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbep4M4/build/debug/bin/kudu
/tmp/dist-test-taskbep4M4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786501898713699-14748-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786501898713699-14748-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786501898713699-14748-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786501898713699-14748-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.103.3:0
--local_ip_for_outbound_sockets=127.14.103.3
--webserver_interface=127.14.103.3
--webserver_port=0
--tserver_master_addrs=127.14.103.62:46141
--builtin_ntp_servers=127.14.103.20:33535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260812 02:36:20.850104 4871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:20.841099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.2:51365 (local address 127.14.103.62:46141)
0812 02:36:20.841250 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0812 02:36:20.841253 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 02:36:20.842356 (+ 1103us) server_negotiation.cc:407] Connection header received
0812 02:36:20.844272 (+ 1916us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:36:20.844276 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:36:20.844326 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:36:20.844420 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:36:20.845666 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:20.846255 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:20.847897 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:20.848290 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:20.848416 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:36:20.849607 (+ 1191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:36:20.849610 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:36:20.849612 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:36:20.849625 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:36:20.849645 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:36:20.849653 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:36:20.849707 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0812 02:36:20.849921 (+ 214us) server_negotiation.cc:299] Negotiation successful
0812 02:36:20.849979 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260812 02:36:20.851385 5198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:20.841515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:46141 (local address 127.14.103.2:51365)
0812 02:36:20.842191 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0812 02:36:20.842233 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0812 02:36:20.844064 (+ 1831us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:36:20.844648 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:36:20.844659 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:36:20.844768 (+ 109us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:36:20.845490 (+ 722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:36:20.845512 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:36:20.846392 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:36:20.846399 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:36:20.847743 (+ 1344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:36:20.847754 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:36:20.848460 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:36:20.848463 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:36:20.848606 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:36:20.849265 (+ 659us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 02:36:20.849269 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:36:20.849350 (+ 81us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 02:36:20.849435 (+ 85us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:36:20.849731 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:36:20.849742 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:36:20.849751 (+ 9us) client_negotiation.cc:769] Sending connection context
0812 02:36:20.849881 (+ 130us) client_negotiation.cc:240] Negotiation successful
0812 02:36:20.850190 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":121,"threads_started":1}
I20260812 02:36:20.852674 5196 heartbeater.cc:347] Connected to a master server at 127.14.103.62:46141
I20260812 02:36:20.852931 5196 heartbeater.cc:464] Registering TS with master...
I20260812 02:36:20.853447 5196 heartbeater.cc:511] Master 127.14.103.62:46141 requested a full tablet report, sending...
I20260812 02:36:20.854681 4821 ts_manager.cc:194] Registered new tserver with Master: ac824397231e44d3ac7ac89e7b8e83a5 (127.14.103.2:41515)
I20260812 02:36:20.855360 4821 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.103.2:51365
W20260812 02:36:20.960573 5199 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:36:20.960860 5199 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:36:20.960901 5199 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:36:20.965065 5199 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:36:20.965133 5199 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:36:20.965191 5199 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.103.3
I20260812 02:36:20.970489 5199 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.103.20:33535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786501898713699-14748-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786501898713699-14748-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.103.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786501898713699-14748-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.103.3
--webserver_port=0
--tserver_master_addrs=127.14.103.62:46141
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5199
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.103.3
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786501898713699-14748-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:36:20.971694 5199 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:36:20.973160 5199 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:36:20.983757 5204 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:36:20.983860 5208 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:36:20.983865 5205 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:36:20.984134 5199 server_base.cc:1034] running on GCE node
I20260812 02:36:20.984570 5199 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:36:20.985313 5199 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:36:20.986490 5199 hybrid_clock.cc:648] HybridClock initialized: now 1786502180986470 us; error 32 us; skew 500 ppm
I20260812 02:36:20.988512 5199 webserver.cc:460] Webserver started at http://127.14.103.3:45745/ using document root <none> and password file <none>
I20260812 02:36:20.989114 5199 fs_manager.cc:359] Metadata directory not provided
I20260812 02:36:20.989178 5199 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:36:20.989396 5199 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:36:20.991611 5199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786501898713699-14748-0/minicluster-data/ts-2/data/instance:
uuid: "65bc79514d5745708147b5ed3d5ed4bc"
format_stamp: "Formatted at 2026-08-12 02:36:20 on dist-test-slave-2w3w"
server_key: "366b1afd123816b994d67f712978eb8d"
server_key_iv: "73428c7b34ffb4c6dcbaa4eb57a31b23"
server_key_version: "encryptionkey@0"
I20260812 02:36:20.992202 5199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786501898713699-14748-0/minicluster-data/ts-2/wal/instance:
uuid: "65bc79514d5745708147b5ed3d5ed4bc"
format_stamp: "Formatted at 2026-08-12 02:36:20 on dist-test-slave-2w3w"
server_key: "366b1afd123816b994d67f712978eb8d"
server_key_iv: "73428c7b34ffb4c6dcbaa4eb57a31b23"
server_key_version: "encryptionkey@0"
I20260812 02:36:20.996075 5199 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260812 02:36:20.998833 5215 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:36:21.000247 5199 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 02:36:21.000393 5199 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786501898713699-14748-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786501898713699-14748-0/minicluster-data/ts-2/wal
uuid: "65bc79514d5745708147b5ed3d5ed4bc"
format_stamp: "Formatted at 2026-08-12 02:36:20 on dist-test-slave-2w3w"
server_key: "366b1afd123816b994d67f712978eb8d"
server_key_iv: "73428c7b34ffb4c6dcbaa4eb57a31b23"
server_key_version: "encryptionkey@0"
I20260812 02:36:21.000520 5199 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786501898713699-14748-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786501898713699-14748-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786501898713699-14748-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:36:21.028713 5199 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:36:21.029479 5199 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:36:21.029690 5199 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:36:21.030313 5199 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:36:21.031380 5199 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:36:21.031450 5199 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:36:21.031522 5199 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:36:21.031561 5199 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:36:21.041483 5199 rpc_server.cc:307] RPC server started. Bound to: 127.14.103.3:36579
I20260812 02:36:21.041509 5344 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.103.3:36579 every 8 connection(s)
I20260812 02:36:21.042418 5199 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786501898713699-14748-0/minicluster-data/ts-2/data/info.pb
I20260812 02:36:21.044106 14748 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbep4M4/build/debug/bin/kudu as pid 5199
I20260812 02:36:21.044250 14748 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786501898713699-14748-0/minicluster-data/ts-2/wal/instance
I20260812 02:36:21.044571 14748 external_mini_cluster.cc:1428] Setting key 1c4130d738123c93befc555b0352c1a7
I20260812 02:36:21.054809 4871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:21.044266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.3:46293 (local address 127.14.103.62:46141)
0812 02:36:21.045000 (+ 734us) server_negotiation.cc:206] Beginning negotiation
0812 02:36:21.045007 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 02:36:21.045575 (+ 568us) server_negotiation.cc:407] Connection header received
0812 02:36:21.047734 (+ 2159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:36:21.047737 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:36:21.047816 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:36:21.047920 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:36:21.049193 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:21.050088 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:21.051761 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:21.052040 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:21.052128 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:36:21.054211 (+ 2083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:36:21.054215 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:36:21.054218 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:36:21.054237 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:36:21.054263 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:36:21.054274 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:36:21.054351 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0812 02:36:21.054551 (+ 200us) server_negotiation.cc:299] Negotiation successful
0812 02:36:21.054622 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":627}
I20260812 02:36:21.055843 5347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:21.044710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:46141 (local address 127.14.103.3:46293)
0812 02:36:21.045423 (+ 713us) negotiation.cc:107] Waiting for socket to connect
0812 02:36:21.045476 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0812 02:36:21.047500 (+ 2024us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:36:21.048138 (+ 638us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:36:21.048152 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:36:21.048272 (+ 120us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:36:21.049026 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:36:21.049047 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:36:21.050253 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:36:21.050256 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:36:21.051579 (+ 1323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:36:21.051591 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:36:21.052297 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:36:21.052302 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:36:21.052552 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:36:21.053810 (+ 1258us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 02:36:21.053812 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:36:21.053879 (+ 67us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 02:36:21.053979 (+ 100us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:36:21.054377 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:36:21.054387 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:36:21.054395 (+ 8us) client_negotiation.cc:769] Sending connection context
0812 02:36:21.054524 (+ 129us) client_negotiation.cc:240] Negotiation successful
0812 02:36:21.054794 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":129,"threads_started":1}
I20260812 02:36:21.057124 5345 heartbeater.cc:347] Connected to a master server at 127.14.103.62:46141
I20260812 02:36:21.057397 5345 heartbeater.cc:464] Registering TS with master...
I20260812 02:36:21.058056 5345 heartbeater.cc:511] Master 127.14.103.62:46141 requested a full tablet report, sending...
I20260812 02:36:21.059135 4821 ts_manager.cc:194] Registered new tserver with Master: 65bc79514d5745708147b5ed3d5ed4bc (127.14.103.3:36579)
I20260812 02:36:21.059666 4821 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.103.3:46293
I20260812 02:36:21.070750 14748 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 02:36:21.076737 4871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:21.072927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41306 (local address 127.14.103.62:46141)
0812 02:36:21.073068 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0812 02:36:21.073072 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:36:21.073208 (+ 136us) server_negotiation.cc:407] Connection header received
0812 02:36:21.073321 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:36:21.073323 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:36:21.073366 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:36:21.073458 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:36:21.073711 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:21.074244 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:21.075205 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:21.075441 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:21.075533 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:36:21.076375 (+ 842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:36:21.076378 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:36:21.076380 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:36:21.076391 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:36:21.076409 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:36:21.076417 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:36:21.076477 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0812 02:36:21.076567 (+ 90us) server_negotiation.cc:299] Negotiation successful
0812 02:36:21.076621 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260812 02:36:21.079864 4821 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41306:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 02:36:21.084198 4821 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 02:36:21.099654 5357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:21.093329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.2:41515 (local address 127.0.0.1:35036)
0812 02:36:21.094225 (+ 896us) negotiation.cc:107] Waiting for socket to connect
0812 02:36:21.094283 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0812 02:36:21.094460 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:36:21.095241 (+ 781us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:36:21.095244 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:36:21.095257 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:36:21.095352 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:36:21.095359 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:36:21.096722 (+ 1363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:36:21.096725 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:36:21.097948 (+ 1223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:36:21.097957 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:36:21.099049 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:36:21.099056 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:36:21.099177 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:36:21.099218 (+ 41us) client_negotiation.cc:769] Sending connection context
0812 02:36:21.099342 (+ 124us) client_negotiation.cc:240] Negotiation successful
0812 02:36:21.099459 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":698,"mutex_wait_us":51,"spinlock_wait_cycles":7680,"thread_start_us":117,"threads_started":1}
I20260812 02:36:21.099956 5361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:21.094660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.3:36579 (local address 127.0.0.1:54024)
0812 02:36:21.094904 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0812 02:36:21.094920 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 02:36:21.095111 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:36:21.095786 (+ 675us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:36:21.095788 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:36:21.095806 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:36:21.095893 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:36:21.095899 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:36:21.097237 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:36:21.097240 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:36:21.098462 (+ 1222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:36:21.098472 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:36:21.099561 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:36:21.099565 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:36:21.099644 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:36:21.099670 (+ 26us) client_negotiation.cc:769] Sending connection context
0812 02:36:21.099749 (+ 79us) client_negotiation.cc:240] Negotiation successful
0812 02:36:21.099823 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"mutex_wait_us":20,"thread_start_us":121,"threads_started":1}
I20260812 02:36:21.100040 5359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:21.093658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35036 (local address 127.14.103.2:41515)
0812 02:36:21.094190 (+ 532us) server_negotiation.cc:206] Beginning negotiation
0812 02:36:21.094206 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0812 02:36:21.094421 (+ 215us) server_negotiation.cc:407] Connection header received
0812 02:36:21.094665 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:36:21.094671 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:36:21.094899 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:36:21.095083 (+ 184us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:36:21.095504 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:21.096583 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:21.098081 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:21.098842 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:21.098984 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:36:21.099197 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0812 02:36:21.099757 (+ 560us) server_negotiation.cc:299] Negotiation successful
0812 02:36:21.099886 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":437,"thread_start_us":155,"threads_started":1}
I20260812 02:36:21.100185 5356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:21.093327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.1:41463 (local address 127.0.0.1:59330)
0812 02:36:21.094289 (+ 962us) negotiation.cc:107] Waiting for socket to connect
0812 02:36:21.094304 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 02:36:21.094460 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:36:21.095127 (+ 667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:36:21.095133 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:36:21.095157 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:36:21.095525 (+ 368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:36:21.095534 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:36:21.096641 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:36:21.096646 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:36:21.098071 (+ 1425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:36:21.098081 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:36:21.099306 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:36:21.099309 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:36:21.099415 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:36:21.099442 (+ 27us) client_negotiation.cc:769] Sending connection context
0812 02:36:21.099527 (+ 85us) client_negotiation.cc:240] Negotiation successful
0812 02:36:21.099601 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":903,"mutex_wait_us":437,"spinlock_wait_cycles":1280,"thread_start_us":118,"threads_started":1}
I20260812 02:36:21.100687 5358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:21.093493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59330 (local address 127.14.103.1:41463)
0812 02:36:21.094230 (+ 737us) server_negotiation.cc:206] Beginning negotiation
0812 02:36:21.094243 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0812 02:36:21.094609 (+ 366us) server_negotiation.cc:407] Connection header received
0812 02:36:21.094682 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:36:21.094687 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:36:21.094902 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:36:21.095363 (+ 461us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:36:21.095642 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:21.096525 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:21.098216 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:21.098931 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:21.099100 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:36:21.099305 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0812 02:36:21.100461 (+ 1156us) server_negotiation.cc:299] Negotiation successful
0812 02:36:21.100565 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":619,"thread_start_us":128,"threads_started":1}
I20260812 02:36:21.100690 5360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:21.094292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54024 (local address 127.14.103.3:36579)
0812 02:36:21.094621 (+ 329us) server_negotiation.cc:206] Beginning negotiation
0812 02:36:21.094632 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 02:36:21.095037 (+ 405us) server_negotiation.cc:407] Connection header received
0812 02:36:21.095262 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:36:21.095269 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:36:21.095467 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:36:21.095593 (+ 126us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:36:21.096026 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:21.097120 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:21.098593 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:21.099422 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:21.099547 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:36:21.099725 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0812 02:36:21.100428 (+ 703us) server_negotiation.cc:299] Negotiation successful
0812 02:36:21.100538 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":237,"thread_start_us":141,"threads_started":1}
I20260812 02:36:21.103047 5121 tablet_service.cc:1467] Processing CreateTablet for tablet 81a033a039cd44bfb21878591ae656f9 (DEFAULT_TABLE table=test-table [id=54b83ec1b1564b82930b94905a3b3fc2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:36:21.103319 5266 tablet_service.cc:1467] Processing CreateTablet for tablet 81a033a039cd44bfb21878591ae656f9 (DEFAULT_TABLE table=test-table [id=54b83ec1b1564b82930b94905a3b3fc2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:36:21.103310 4969 tablet_service.cc:1467] Processing CreateTablet for tablet 81a033a039cd44bfb21878591ae656f9 (DEFAULT_TABLE table=test-table [id=54b83ec1b1564b82930b94905a3b3fc2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:36:21.104610 4969 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81a033a039cd44bfb21878591ae656f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:36:21.104610 5266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81a033a039cd44bfb21878591ae656f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:36:21.104656 5121 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81a033a039cd44bfb21878591ae656f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:36:21.111735 5362 tablet_bootstrap.cc:492] T 81a033a039cd44bfb21878591ae656f9 P ac824397231e44d3ac7ac89e7b8e83a5: Bootstrap starting.
I20260812 02:36:21.113058 5363 tablet_bootstrap.cc:492] T 81a033a039cd44bfb21878591ae656f9 P fb1cbd7f4a1841dc8a34b2bdcc615e14: Bootstrap starting.
I20260812 02:36:21.113046 5364 tablet_bootstrap.cc:492] T 81a033a039cd44bfb21878591ae656f9 P 65bc79514d5745708147b5ed3d5ed4bc: Bootstrap starting.
I20260812 02:36:21.114038 5362 tablet_bootstrap.cc:654] T 81a033a039cd44bfb21878591ae656f9 P ac824397231e44d3ac7ac89e7b8e83a5: Neither blocks nor log segments found. Creating new log.
I20260812 02:36:21.114809 5362 log.cc:824] T 81a033a039cd44bfb21878591ae656f9 P ac824397231e44d3ac7ac89e7b8e83a5: Log is configured to *not* fsync() on all Append() calls
I20260812 02:36:21.115182 5364 tablet_bootstrap.cc:654] T 81a033a039cd44bfb21878591ae656f9 P 65bc79514d5745708147b5ed3d5ed4bc: Neither blocks nor log segments found. Creating new log.
I20260812 02:36:21.115182 5363 tablet_bootstrap.cc:654] T 81a033a039cd44bfb21878591ae656f9 P fb1cbd7f4a1841dc8a34b2bdcc615e14: Neither blocks nor log segments found. Creating new log.
I20260812 02:36:21.116065 5363 log.cc:824] T 81a033a039cd44bfb21878591ae656f9 P fb1cbd7f4a1841dc8a34b2bdcc615e14: Log is configured to *not* fsync() on all Append() calls
I20260812 02:36:21.116056 5364 log.cc:824] T 81a033a039cd44bfb21878591ae656f9 P 65bc79514d5745708147b5ed3d5ed4bc: Log is configured to *not* fsync() on all Append() calls
I20260812 02:36:21.117807 5362 tablet_bootstrap.cc:492] T 81a033a039cd44bfb21878591ae656f9 P ac824397231e44d3ac7ac89e7b8e83a5: No bootstrap required, opened a new log
I20260812 02:36:21.118017 5362 ts_tablet_manager.cc:1397] T 81a033a039cd44bfb21878591ae656f9 P ac824397231e44d3ac7ac89e7b8e83a5: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260812 02:36:21.118069 5364 tablet_bootstrap.cc:492] T 81a033a039cd44bfb21878591ae656f9 P 65bc79514d5745708147b5ed3d5ed4bc: No bootstrap required, opened a new log
I20260812 02:36:21.118227 5364 ts_tablet_manager.cc:1397] T 81a033a039cd44bfb21878591ae656f9 P 65bc79514d5745708147b5ed3d5ed4bc: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260812 02:36:21.119917 5363 tablet_bootstrap.cc:492] T 81a033a039cd44bfb21878591ae656f9 P fb1cbd7f4a1841dc8a34b2bdcc615e14: No bootstrap required, opened a new log
I20260812 02:36:21.120112 5363 ts_tablet_manager.cc:1397] T 81a033a039cd44bfb21878591ae656f9 P fb1cbd7f4a1841dc8a34b2bdcc615e14: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20260812 02:36:21.122205 5362 raft_consensus.cc:348] T 81a033a039cd44bfb21878591ae656f9 P ac824397231e44d3ac7ac89e7b8e83a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb1cbd7f4a1841dc8a34b2bdcc615e14" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 41463 } } peers { permanent_uuid: "ac824397231e44d3ac7ac89e7b8e83a5" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 41515 } } peers { permanent_uuid: "65bc79514d5745708147b5ed3d5ed4bc" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 36579 } }
I20260812 02:36:21.122308 5364 raft_consensus.cc:348] T 81a033a039cd44bfb21878591ae656f9 P 65bc79514d5745708147b5ed3d5ed4bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb1cbd7f4a1841dc8a34b2bdcc615e14" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 41463 } } peers { permanent_uuid: "ac824397231e44d3ac7ac89e7b8e83a5" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 41515 } } peers { permanent_uuid: "65bc79514d5745708147b5ed3d5ed4bc" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 36579 } }
I20260812 02:36:21.122505 5362 raft_consensus.cc:374] T 81a033a039cd44bfb21878591ae656f9 P ac824397231e44d3ac7ac89e7b8e83a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:36:21.122505 5364 raft_consensus.cc:374] T 81a033a039cd44bfb21878591ae656f9 P 65bc79514d5745708147b5ed3d5ed4bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:36:21.122565 5362 raft_consensus.cc:729] T 81a033a039cd44bfb21878591ae656f9 P ac824397231e44d3ac7ac89e7b8e83a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac824397231e44d3ac7ac89e7b8e83a5, State: Initialized, Role: FOLLOWER
I20260812 02:36:21.122565 5364 raft_consensus.cc:729] T 81a033a039cd44bfb21878591ae656f9 P 65bc79514d5745708147b5ed3d5ed4bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65bc79514d5745708147b5ed3d5ed4bc, State: Initialized, Role: FOLLOWER
I20260812 02:36:21.123076 5362 consensus_queue.cc:260] T 81a033a039cd44bfb21878591ae656f9 P ac824397231e44d3ac7ac89e7b8e83a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb1cbd7f4a1841dc8a34b2bdcc615e14" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 41463 } } peers { permanent_uuid: "ac824397231e44d3ac7ac89e7b8e83a5" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 41515 } } peers { permanent_uuid: "65bc79514d5745708147b5ed3d5ed4bc" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 36579 } }
I20260812 02:36:21.123145 5364 consensus_queue.cc:260] T 81a033a039cd44bfb21878591ae656f9 P 65bc79514d5745708147b5ed3d5ed4bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb1cbd7f4a1841dc8a34b2bdcc615e14" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 41463 } } peers { permanent_uuid: "ac824397231e44d3ac7ac89e7b8e83a5" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 41515 } } peers { permanent_uuid: "65bc79514d5745708147b5ed3d5ed4bc" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 36579 } }
I20260812 02:36:21.123507 5363 raft_consensus.cc:348] T 81a033a039cd44bfb21878591ae656f9 P fb1cbd7f4a1841dc8a34b2bdcc615e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb1cbd7f4a1841dc8a34b2bdcc615e14" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 41463 } } peers { permanent_uuid: "ac824397231e44d3ac7ac89e7b8e83a5" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 41515 } } peers { permanent_uuid: "65bc79514d5745708147b5ed3d5ed4bc" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 36579 } }
I20260812 02:36:21.123764 5363 raft_consensus.cc:374] T 81a033a039cd44bfb21878591ae656f9 P fb1cbd7f4a1841dc8a34b2bdcc615e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:36:21.123831 5363 raft_consensus.cc:729] T 81a033a039cd44bfb21878591ae656f9 P fb1cbd7f4a1841dc8a34b2bdcc615e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb1cbd7f4a1841dc8a34b2bdcc615e14, State: Initialized, Role: FOLLOWER
I20260812 02:36:21.124146 5364 ts_tablet_manager.cc:1428] T 81a033a039cd44bfb21878591ae656f9 P 65bc79514d5745708147b5ed3d5ed4bc: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260812 02:36:21.124146 5362 ts_tablet_manager.cc:1428] T 81a033a039cd44bfb21878591ae656f9 P ac824397231e44d3ac7ac89e7b8e83a5: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260812 02:36:21.124271 5363 consensus_queue.cc:260] T 81a033a039cd44bfb21878591ae656f9 P fb1cbd7f4a1841dc8a34b2bdcc615e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb1cbd7f4a1841dc8a34b2bdcc615e14" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 41463 } } peers { permanent_uuid: "ac824397231e44d3ac7ac89e7b8e83a5" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 41515 } } peers { permanent_uuid: "65bc79514d5745708147b5ed3d5ed4bc" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 36579 } }
I20260812 02:36:21.124512 5345 heartbeater.cc:503] Master 127.14.103.62:46141 was elected leader, sending a full tablet report...
I20260812 02:36:21.124934 5046 heartbeater.cc:503] Master 127.14.103.62:46141 was elected leader, sending a full tablet report...
I20260812 02:36:21.125200 5363 ts_tablet_manager.cc:1428] T 81a033a039cd44bfb21878591ae656f9 P fb1cbd7f4a1841dc8a34b2bdcc615e14: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260812 02:36:21.125614 5196 heartbeater.cc:503] Master 127.14.103.62:46141 was elected leader, sending a full tablet report...
W20260812 02:36:21.152189 5048 tablet.cc:2369] T 81a033a039cd44bfb21878591ae656f9 P fb1cbd7f4a1841dc8a34b2bdcc615e14: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 02:36:21.180354 5372 raft_consensus.cc:482] T 81a033a039cd44bfb21878591ae656f9 P 65bc79514d5745708147b5ed3d5ed4bc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 02:36:21.180573 5372 raft_consensus.cc:504] T 81a033a039cd44bfb21878591ae656f9 P 65bc79514d5745708147b5ed3d5ed4bc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb1cbd7f4a1841dc8a34b2bdcc615e14" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 41463 } } peers { permanent_uuid: "ac824397231e44d3ac7ac89e7b8e83a5" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 41515 } } peers { permanent_uuid: "65bc79514d5745708147b5ed3d5ed4bc" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 36579 } }
I20260812 02:36:21.181784 5372 leader_election.cc:288] T 81a033a039cd44bfb21878591ae656f9 P 65bc79514d5745708147b5ed3d5ed4bc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fb1cbd7f4a1841dc8a34b2bdcc615e14 (127.14.103.1:41463), ac824397231e44d3ac7ac89e7b8e83a5 (127.14.103.2:41515)
I20260812 02:36:21.187533 5359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:21.182256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.3:45991 (local address 127.14.103.2:41515)
0812 02:36:21.182392 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0812 02:36:21.182397 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:36:21.182668 (+ 271us) server_negotiation.cc:407] Connection header received
0812 02:36:21.182732 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:36:21.182735 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:36:21.182794 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:36:21.182873 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:36:21.183369 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:21.184204 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:21.186077 (+ 1873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:21.186687 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:21.186858 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:36:21.186899 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0812 02:36:21.187300 (+ 401us) server_negotiation.cc:299] Negotiation successful
0812 02:36:21.187388 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260812 02:36:21.187533 5358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:21.182313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.103.3:37279 (local address 127.14.103.1:41463)
0812 02:36:21.182442 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0812 02:36:21.182452 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 02:36:21.182467 (+ 15us) server_negotiation.cc:407] Connection header received
0812 02:36:21.182672 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:36:21.182676 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:36:21.182740 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:36:21.182845 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:36:21.183393 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:21.184162 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:21.186077 (+ 1915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:21.186687 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:21.186842 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:36:21.186899 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0812 02:36:21.187300 (+ 401us) server_negotiation.cc:299] Negotiation successful
0812 02:36:21.187389 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260812 02:36:21.187567 5347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:21.182076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.1:41463 (local address 127.14.103.3:37279)
0812 02:36:21.182357 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0812 02:36:21.182376 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 02:36:21.182479 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:36:21.182911 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:36:21.182915 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:36:21.182959 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:36:21.183201 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:36:21.183208 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:36:21.184334 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:36:21.184338 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:36:21.185884 (+ 1546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:36:21.185898 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:36:21.186859 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:36:21.186864 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:36:21.187064 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:36:21.187121 (+ 57us) client_negotiation.cc:769] Sending connection context
0812 02:36:21.187209 (+ 88us) client_negotiation.cc:240] Negotiation successful
0812 02:36:21.187388 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"mutex_wait_us":229,"spinlock_wait_cycles":32512}
I20260812 02:36:21.187577 5374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:21.182076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.2:41515 (local address 127.14.103.3:45991)
0812 02:36:21.182544 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0812 02:36:21.182558 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 02:36:21.182646 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:36:21.182911 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:36:21.182915 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:36:21.182984 (+ 69us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:36:21.183199 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:36:21.183208 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:36:21.184334 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:36:21.184338 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:36:21.185928 (+ 1590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:36:21.185936 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:36:21.186859 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:36:21.186864 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:36:21.187043 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:36:21.187123 (+ 80us) client_negotiation.cc:769] Sending connection context
0812 02:36:21.187209 (+ 86us) client_negotiation.cc:240] Negotiation successful
0812 02:36:21.187388 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":382,"mutex_wait_us":378,"spinlock_wait_cycles":2688,"thread_start_us":136,"threads_started":1}
I20260812 02:36:21.188292 4992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81a033a039cd44bfb21878591ae656f9" candidate_uuid: "65bc79514d5745708147b5ed3d5ed4bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb1cbd7f4a1841dc8a34b2bdcc615e14" is_pre_election: true
I20260812 02:36:21.188512 5146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81a033a039cd44bfb21878591ae656f9" candidate_uuid: "65bc79514d5745708147b5ed3d5ed4bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac824397231e44d3ac7ac89e7b8e83a5" is_pre_election: true
I20260812 02:36:21.188624 4992 raft_consensus.cc:2454] T 81a033a039cd44bfb21878591ae656f9 P fb1cbd7f4a1841dc8a34b2bdcc615e14 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65bc79514d5745708147b5ed3d5ed4bc in term 0.
I20260812 02:36:21.188808 5146 raft_consensus.cc:2454] T 81a033a039cd44bfb21878591ae656f9 P ac824397231e44d3ac7ac89e7b8e83a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65bc79514d5745708147b5ed3d5ed4bc in term 0.
I20260812 02:36:21.189235 5216 leader_election.cc:302] T 81a033a039cd44bfb21878591ae656f9 P 65bc79514d5745708147b5ed3d5ed4bc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65bc79514d5745708147b5ed3d5ed4bc, fb1cbd7f4a1841dc8a34b2bdcc615e14; no voters:
I20260812 02:36:21.189554 5372 raft_consensus.cc:2781] T 81a033a039cd44bfb21878591ae656f9 P 65bc79514d5745708147b5ed3d5ed4bc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 02:36:21.189651 5372 raft_consensus.cc:482] T 81a033a039cd44bfb21878591ae656f9 P 65bc79514d5745708147b5ed3d5ed4bc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 02:36:21.189694 5372 raft_consensus.cc:3037] T 81a033a039cd44bfb21878591ae656f9 P 65bc79514d5745708147b5ed3d5ed4bc [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:36:21.190950 5372 raft_consensus.cc:504] T 81a033a039cd44bfb21878591ae656f9 P 65bc79514d5745708147b5ed3d5ed4bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb1cbd7f4a1841dc8a34b2bdcc615e14" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 41463 } } peers { permanent_uuid: "ac824397231e44d3ac7ac89e7b8e83a5" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 41515 } } peers { permanent_uuid: "65bc79514d5745708147b5ed3d5ed4bc" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 36579 } }
I20260812 02:36:21.191434 5372 leader_election.cc:288] T 81a033a039cd44bfb21878591ae656f9 P 65bc79514d5745708147b5ed3d5ed4bc [CANDIDATE]: Term 1 election: Requested vote from peers fb1cbd7f4a1841dc8a34b2bdcc615e14 (127.14.103.1:41463), ac824397231e44d3ac7ac89e7b8e83a5 (127.14.103.2:41515)
I20260812 02:36:21.191857 5146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81a033a039cd44bfb21878591ae656f9" candidate_uuid: "65bc79514d5745708147b5ed3d5ed4bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac824397231e44d3ac7ac89e7b8e83a5"
I20260812 02:36:21.191857 4992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81a033a039cd44bfb21878591ae656f9" candidate_uuid: "65bc79514d5745708147b5ed3d5ed4bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb1cbd7f4a1841dc8a34b2bdcc615e14"
I20260812 02:36:21.191998 4992 raft_consensus.cc:3037] T 81a033a039cd44bfb21878591ae656f9 P fb1cbd7f4a1841dc8a34b2bdcc615e14 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:36:21.191998 5146 raft_consensus.cc:3037] T 81a033a039cd44bfb21878591ae656f9 P ac824397231e44d3ac7ac89e7b8e83a5 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:36:21.193430 4992 raft_consensus.cc:2454] T 81a033a039cd44bfb21878591ae656f9 P fb1cbd7f4a1841dc8a34b2bdcc615e14 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65bc79514d5745708147b5ed3d5ed4bc in term 1.
I20260812 02:36:21.193430 5146 raft_consensus.cc:2454] T 81a033a039cd44bfb21878591ae656f9 P ac824397231e44d3ac7ac89e7b8e83a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65bc79514d5745708147b5ed3d5ed4bc in term 1.
I20260812 02:36:21.193827 5216 leader_election.cc:302] T 81a033a039cd44bfb21878591ae656f9 P 65bc79514d5745708147b5ed3d5ed4bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 65bc79514d5745708147b5ed3d5ed4bc, ac824397231e44d3ac7ac89e7b8e83a5, fb1cbd7f4a1841dc8a34b2bdcc615e14; no voters:
I20260812 02:36:21.194034 5372 raft_consensus.cc:2781] T 81a033a039cd44bfb21878591ae656f9 P 65bc79514d5745708147b5ed3d5ed4bc [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:36:21.194283 5372 raft_consensus.cc:686] T 81a033a039cd44bfb21878591ae656f9 P 65bc79514d5745708147b5ed3d5ed4bc [term 1 LEADER]: Becoming Leader. State: Replica: 65bc79514d5745708147b5ed3d5ed4bc, State: Running, Role: LEADER
I20260812 02:36:21.194617 5372 consensus_queue.cc:237] T 81a033a039cd44bfb21878591ae656f9 P 65bc79514d5745708147b5ed3d5ed4bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb1cbd7f4a1841dc8a34b2bdcc615e14" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 41463 } } peers { permanent_uuid: "ac824397231e44d3ac7ac89e7b8e83a5" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 41515 } } peers { permanent_uuid: "65bc79514d5745708147b5ed3d5ed4bc" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 36579 } }
I20260812 02:36:21.198072 4818 catalog_manager.cc:5495] T 81a033a039cd44bfb21878591ae656f9 P 65bc79514d5745708147b5ed3d5ed4bc reported cstate change: term changed from 0 to 1, leader changed from <none> to 65bc79514d5745708147b5ed3d5ed4bc (127.14.103.3). New cstate: current_term: 1 leader_uuid: "65bc79514d5745708147b5ed3d5ed4bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb1cbd7f4a1841dc8a34b2bdcc615e14" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 41463 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ac824397231e44d3ac7ac89e7b8e83a5" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 41515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65bc79514d5745708147b5ed3d5ed4bc" member_type: VOTER last_known_addr { host: "127.14.103.3" port: 36579 } health_report { overall_health: HEALTHY } } }
I20260812 02:36:21.224751 5360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:21.221044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54040 (local address 127.14.103.3:36579)
0812 02:36:21.221190 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0812 02:36:21.221195 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:36:21.221213 (+ 18us) server_negotiation.cc:407] Connection header received
0812 02:36:21.221314 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:36:21.221318 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:36:21.221375 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:36:21.221491 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 02:36:21.221739 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:21.222293 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:21.223240 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:21.223472 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:21.223583 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:36:21.223900 (+ 317us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 02:36:21.224342 (+ 442us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 02:36:21.224481 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0812 02:36:21.224543 (+ 62us) server_negotiation.cc:299] Negotiation successful
0812 02:36:21.224599 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260812 02:36:21.230801 4992 raft_consensus.cc:1261] T 81a033a039cd44bfb21878591ae656f9 P fb1cbd7f4a1841dc8a34b2bdcc615e14 [term 1 FOLLOWER]: Refusing update from remote peer 65bc79514d5745708147b5ed3d5ed4bc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:36:21.230800 5146 raft_consensus.cc:1261] T 81a033a039cd44bfb21878591ae656f9 P ac824397231e44d3ac7ac89e7b8e83a5 [term 1 FOLLOWER]: Refusing update from remote peer 65bc79514d5745708147b5ed3d5ed4bc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:36:21.231607 5375 consensus_queue.cc:1035] T 81a033a039cd44bfb21878591ae656f9 P 65bc79514d5745708147b5ed3d5ed4bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac824397231e44d3ac7ac89e7b8e83a5" member_type: VOTER last_known_addr { host: "127.14.103.2" port: 41515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:36:21.231813 5372 consensus_queue.cc:1035] T 81a033a039cd44bfb21878591ae656f9 P 65bc79514d5745708147b5ed3d5ed4bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb1cbd7f4a1841dc8a34b2bdcc615e14" member_type: VOTER last_known_addr { host: "127.14.103.1" port: 41463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:36:21.239990 5379 mvcc.cc:205] Tried to move back new op lower bound from 7317512934313779200 to 7317512934176243712. Current Snapshot: MvccSnapshot[applied={T|T < 7317512934313779200}]
I20260812 02:36:21.240031 5380 mvcc.cc:205] Tried to move back new op lower bound from 7317512934313779200 to 7317512934176243712. Current Snapshot: MvccSnapshot[applied={T|T < 7317512934313779200}]
I20260812 02:36:21.248054 4818 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41306:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 02:36:21.248219 4818 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41306:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 02:36:21.250612 4818 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 12f7cf80816147a894b04155e4ab7032: Sending DeleteTablet for 3 replicas of tablet 81a033a039cd44bfb21878591ae656f9
I20260812 02:36:21.251503 4969 tablet_service.cc:1514] Processing DeleteTablet for tablet 81a033a039cd44bfb21878591ae656f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:36:21 UTC) from {username='slave'} at 127.0.0.1:59330
I20260812 02:36:21.251566 5266 tablet_service.cc:1514] Processing DeleteTablet for tablet 81a033a039cd44bfb21878591ae656f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:36:21 UTC) from {username='slave'} at 127.0.0.1:54024
I20260812 02:36:21.251612 5121 tablet_service.cc:1514] Processing DeleteTablet for tablet 81a033a039cd44bfb21878591ae656f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:36:21 UTC) from {username='slave'} at 127.0.0.1:35036
I20260812 02:36:21.252085 5389 tablet_replica.cc:330] T 81a033a039cd44bfb21878591ae656f9 P ac824397231e44d3ac7ac89e7b8e83a5: stopping tablet replica
I20260812 02:36:21.252099 5390 tablet_replica.cc:330] T 81a033a039cd44bfb21878591ae656f9 P fb1cbd7f4a1841dc8a34b2bdcc615e14: stopping tablet replica
I20260812 02:36:21.252420 5390 raft_consensus.cc:2229] T 81a033a039cd44bfb21878591ae656f9 P fb1cbd7f4a1841dc8a34b2bdcc615e14 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 02:36:21.252421 5389 raft_consensus.cc:2229] T 81a033a039cd44bfb21878591ae656f9 P ac824397231e44d3ac7ac89e7b8e83a5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 02:36:21.252738 5390 raft_consensus.cc:2258] T 81a033a039cd44bfb21878591ae656f9 P fb1cbd7f4a1841dc8a34b2bdcc615e14 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:36:21.252744 5389 raft_consensus.cc:2258] T 81a033a039cd44bfb21878591ae656f9 P ac824397231e44d3ac7ac89e7b8e83a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:36:21.253055 14748 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbep4M4/build/debug/bin/kudu with pid 4894
I20260812 02:36:21.253134 5391 tablet_replica.cc:330] T 81a033a039cd44bfb21878591ae656f9 P 65bc79514d5745708147b5ed3d5ed4bc: stopping tablet replica
I20260812 02:36:21.254848 5391 raft_consensus.cc:2229] T 81a033a039cd44bfb21878591ae656f9 P 65bc79514d5745708147b5ed3d5ed4bc [term 1 LEADER]: Raft consensus shutting down.
I20260812 02:36:21.255304 5391 raft_consensus.cc:2258] T 81a033a039cd44bfb21878591ae656f9 P 65bc79514d5745708147b5ed3d5ed4bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:36:21.257169 5391 ts_tablet_manager.cc:1905] T 81a033a039cd44bfb21878591ae656f9 P 65bc79514d5745708147b5ed3d5ed4bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:36:21.260159 5391 ts_tablet_manager.cc:1918] T 81a033a039cd44bfb21878591ae656f9 P 65bc79514d5745708147b5ed3d5ed4bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 02:36:21.260250 5391 log.cc:1196] T 81a033a039cd44bfb21878591ae656f9 P 65bc79514d5745708147b5ed3d5ed4bc: Deleting WAL directory at /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786501898713699-14748-0/minicluster-data/ts-2/wal/wals/81a033a039cd44bfb21878591ae656f9
I20260812 02:36:21.260572 5391 ts_tablet_manager.cc:1939] T 81a033a039cd44bfb21878591ae656f9 P 65bc79514d5745708147b5ed3d5ed4bc: Deleting consensus metadata
I20260812 02:36:21.261821 4805 catalog_manager.cc:4841] TS 65bc79514d5745708147b5ed3d5ed4bc (127.14.103.3:36579): tablet 81a033a039cd44bfb21878591ae656f9 (table test-table [id=54b83ec1b1564b82930b94905a3b3fc2]) successfully deleted
I20260812 02:36:21.264137 5389 ts_tablet_manager.cc:1905] T 81a033a039cd44bfb21878591ae656f9 P ac824397231e44d3ac7ac89e7b8e83a5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:36:21.267043 5389 ts_tablet_manager.cc:1918] T 81a033a039cd44bfb21878591ae656f9 P ac824397231e44d3ac7ac89e7b8e83a5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 02:36:21.267192 5389 log.cc:1196] T 81a033a039cd44bfb21878591ae656f9 P ac824397231e44d3ac7ac89e7b8e83a5: Deleting WAL directory at /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786501898713699-14748-0/minicluster-data/ts-1/wal/wals/81a033a039cd44bfb21878591ae656f9
I20260812 02:36:21.267486 5389 ts_tablet_manager.cc:1939] T 81a033a039cd44bfb21878591ae656f9 P ac824397231e44d3ac7ac89e7b8e83a5: Deleting consensus metadata
I20260812 02:36:21.268656 4805 catalog_manager.cc:4841] TS ac824397231e44d3ac7ac89e7b8e83a5 (127.14.103.2:41515): tablet 81a033a039cd44bfb21878591ae656f9 (table test-table [id=54b83ec1b1564b82930b94905a3b3fc2]) successfully deleted
W20260812 02:36:21.269620 4802 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.103.1:41463 (error 108)
I20260812 02:36:21.270509 14748 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbep4M4/build/debug/bin/kudu with pid 5050
I20260812 02:36:21.270697 5356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:21.270374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.1:41463 (local address 127.0.0.1:59334)
0812 02:36:21.270541 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0812 02:36:21.270614 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.1:41463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260812 02:36:21.276597 4802 catalog_manager.cc:4584] TS fb1cbd7f4a1841dc8a34b2bdcc615e14 (127.14.103.1:41463): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 81a033a039cd44bfb21878591ae656f9: Network error: Client connection negotiation failed: client connection to 127.14.103.1:41463: connect: Connection refused (error 111)
I20260812 02:36:21.290076 14748 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbep4M4/build/debug/bin/kudu with pid 5199
I20260812 02:36:21.302161 14748 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbep4M4/build/debug/bin/kudu with pid 4786
2026-08-12T02:36:21Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1162 ms)
[----------] 6 tests from EncryptionPolicyTest (11539 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786501898713699-14748-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 02:36:21 dist-test-slave-2w3w krb5kdc[5397](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 02:36:21 dist-test-slave-2w3w krb5kdc[5397](info): set up 2 sockets
Aug 12 02:36:21 dist-test-slave-2w3w krb5kdc[5397](info): commencing operation
krb5kdc: starting...
W20260812 02:36:22.414402 14748 mini_kdc.cc:121] Time spent starting KDC: real 1.089s user 0.001s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786501898713699-14748-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786501898713699-14748-0/krb5kdc/test-user.keytab.
Aug 12 02:36:22 dist-test-slave-2w3w krb5kdc[5397](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502182, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T02:36:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T02:36:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786501898713699-14748-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786501898713699-14748-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 02:36:22.554162 14748 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbep4M4/build/debug/bin/kudu
/tmp/dist-test-taskbep4M4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786501898713699-14748-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786501898713699-14748-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786501898713699-14748-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786501898713699-14748-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45861
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786501898713699-14748-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45861
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786501898713699-14748-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786501898713699-14748-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786501898713699-14748-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:36:22.668339 5414 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:36:22.668629 5414 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:36:22.668672 5414 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:36:22.672694 5414 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:36:22.672755 5414 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:36:22.672775 5414 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:36:22.672791 5414 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:36:22.672806 5414 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 02:36:22.678139 5414 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786501898713699-14748-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786501898713699-14748-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45861
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786501898713699-14748-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45861
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786501898713699-14748-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5414
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786501898713699-14748-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:36:22.679190 5414 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:36:22.680193 5414 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:36:22.689231 5424 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:36:22.689301 5421 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:36:22.689301 5422 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:36:22.689466 5414 server_base.cc:1034] running on GCE node
I20260812 02:36:22.690192 5414 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:36:22.691474 5414 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:36:22.692684 5414 hybrid_clock.cc:648] HybridClock initialized: now 1786502182692671 us; error 33 us; skew 500 ppm
Aug 12 02:36:22 dist-test-slave-2w3w krb5kdc[5397](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502182, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:36:22.695598 5414 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 02:36:22.696489 5414 webserver.cc:460] Webserver started at http://127.0.0.1:44155/ using document root <none> and password file <none>
I20260812 02:36:22.697013 5414 fs_manager.cc:359] Metadata directory not provided
I20260812 02:36:22.697073 5414 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:36:22.697243 5414 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:36:22.699419 5414 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786501898713699-14748-0/minicluster-data/master-0/data/instance:
uuid: "bc598874cd78452bbfc3c21e26b35aa3"
format_stamp: "Formatted at 2026-08-12 02:36:22 on dist-test-slave-2w3w"
server_key: "fac9744d3225a08cead6ac078830743d"
server_key_iv: "f1534cb736dba9a2476fcc082e2a34f4"
server_key_version: "encryptionkey@0"
I20260812 02:36:22.699950 5414 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786501898713699-14748-0/minicluster-data/master-0/wal/instance:
uuid: "bc598874cd78452bbfc3c21e26b35aa3"
format_stamp: "Formatted at 2026-08-12 02:36:22 on dist-test-slave-2w3w"
server_key: "fac9744d3225a08cead6ac078830743d"
server_key_iv: "f1534cb736dba9a2476fcc082e2a34f4"
server_key_version: "encryptionkey@0"
I20260812 02:36:22.703959 5414 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260812 02:36:22.706736 5432 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:36:22.708087 5414 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260812 02:36:22.708199 5414 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786501898713699-14748-0/minicluster-data/master-0/data,/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786501898713699-14748-0/minicluster-data/master-0/wal
uuid: "bc598874cd78452bbfc3c21e26b35aa3"
format_stamp: "Formatted at 2026-08-12 02:36:22 on dist-test-slave-2w3w"
server_key: "fac9744d3225a08cead6ac078830743d"
server_key_iv: "f1534cb736dba9a2476fcc082e2a34f4"
server_key_version: "encryptionkey@0"
I20260812 02:36:22.708297 5414 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786501898713699-14748-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786501898713699-14748-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786501898713699-14748-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:36:22.733043 5414 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:36:22.736269 5414 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:36:22.736438 5414 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:36:22.744035 5414 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45861
I20260812 02:36:22.744061 5495 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45861 every 8 connection(s)
I20260812 02:36:22.745038 5414 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786501898713699-14748-0/minicluster-data/master-0/data/info.pb
I20260812 02:36:22.747896 5496 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:36:22.751933 14748 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbep4M4/build/debug/bin/kudu as pid 5414
I20260812 02:36:22.752080 14748 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786501898713699-14748-0/minicluster-data/master-0/wal/instance
I20260812 02:36:22.752451 14748 external_mini_cluster.cc:1428] Setting key d0e35e67180f8aa6c0fc862da21a5e17
I20260812 02:36:22.754408 5496 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc598874cd78452bbfc3c21e26b35aa3: Bootstrap starting.
I20260812 02:36:22.756669 5496 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bc598874cd78452bbfc3c21e26b35aa3: Neither blocks nor log segments found. Creating new log.
I20260812 02:36:22.757431 5496 log.cc:824] T 00000000000000000000000000000000 P bc598874cd78452bbfc3c21e26b35aa3: Log is configured to *not* fsync() on all Append() calls
I20260812 02:36:22.759351 5496 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc598874cd78452bbfc3c21e26b35aa3: No bootstrap required, opened a new log
Aug 12 02:36:22 dist-test-slave-2w3w krb5kdc[5397](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502182, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260812 02:36:22.763947 5496 raft_consensus.cc:348] T 00000000000000000000000000000000 P bc598874cd78452bbfc3c21e26b35aa3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc598874cd78452bbfc3c21e26b35aa3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45861 } }
I20260812 02:36:22.764204 5496 raft_consensus.cc:374] T 00000000000000000000000000000000 P bc598874cd78452bbfc3c21e26b35aa3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:36:22.764277 5496 raft_consensus.cc:729] T 00000000000000000000000000000000 P bc598874cd78452bbfc3c21e26b35aa3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc598874cd78452bbfc3c21e26b35aa3, State: Initialized, Role: FOLLOWER
I20260812 02:36:22.764791 5496 consensus_queue.cc:260] T 00000000000000000000000000000000 P bc598874cd78452bbfc3c21e26b35aa3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc598874cd78452bbfc3c21e26b35aa3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45861 } }
I20260812 02:36:22.764983 5496 raft_consensus.cc:388] T 00000000000000000000000000000000 P bc598874cd78452bbfc3c21e26b35aa3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 02:36:22.765058 5496 raft_consensus.cc:482] T 00000000000000000000000000000000 P bc598874cd78452bbfc3c21e26b35aa3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 02:36:22.765180 5496 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bc598874cd78452bbfc3c21e26b35aa3 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:36:22.766338 5496 raft_consensus.cc:504] T 00000000000000000000000000000000 P bc598874cd78452bbfc3c21e26b35aa3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc598874cd78452bbfc3c21e26b35aa3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45861 } }
I20260812 02:36:22.766726 5496 leader_election.cc:302] T 00000000000000000000000000000000 P bc598874cd78452bbfc3c21e26b35aa3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bc598874cd78452bbfc3c21e26b35aa3; no voters:
I20260812 02:36:22.767145 5496 leader_election.cc:288] T 00000000000000000000000000000000 P bc598874cd78452bbfc3c21e26b35aa3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 02:36:22.767273 5503 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bc598874cd78452bbfc3c21e26b35aa3 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:36:22.767556 5503 raft_consensus.cc:686] T 00000000000000000000000000000000 P bc598874cd78452bbfc3c21e26b35aa3 [term 1 LEADER]: Becoming Leader. State: Replica: bc598874cd78452bbfc3c21e26b35aa3, State: Running, Role: LEADER
I20260812 02:36:22.767884 5503 consensus_queue.cc:237] T 00000000000000000000000000000000 P bc598874cd78452bbfc3c21e26b35aa3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc598874cd78452bbfc3c21e26b35aa3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45861 } }
I20260812 02:36:22.768112 5501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:22.753976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49039 (local address 127.0.0.1:45861)
0812 02:36:22.754431 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0812 02:36:22.754447 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0812 02:36:22.754505 (+ 58us) server_negotiation.cc:407] Connection header received
0812 02:36:22.755238 (+ 733us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:36:22.755256 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:36:22.755585 (+ 329us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:36:22.755923 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:36:22.756737 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:22.757834 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:22.758688 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:22.759381 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:22.759579 (+ 198us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:36:22.762578 (+ 2999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:36:22.762583 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:36:22.762597 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:36:22.762626 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:36:22.764613 (+ 1987us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:36:22.765201 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:36:22.765206 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:36:22.765211 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:36:22.765390 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:36:22.765755 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:36:22.765758 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:36:22.765759 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:36:22.766148 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:36:22.766308 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0812 02:36:22.766713 (+ 405us) server_negotiation.cc:299] Negotiation successful
0812 02:36:22.767028 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":145,"threads_started":1}
I20260812 02:36:22.768505 5496 sys_catalog.cc:564] T 00000000000000000000000000000000 P bc598874cd78452bbfc3c21e26b35aa3 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:36:22.769086 5503 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc598874cd78452bbfc3c21e26b35aa3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bc598874cd78452bbfc3c21e26b35aa3. Latest consensus state: current_term: 1 leader_uuid: "bc598874cd78452bbfc3c21e26b35aa3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc598874cd78452bbfc3c21e26b35aa3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45861 } } }
I20260812 02:36:22.769232 5503 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc598874cd78452bbfc3c21e26b35aa3 [sys.catalog]: This master's current role is: LEADER
I20260812 02:36:22.769522 5506 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc598874cd78452bbfc3c21e26b35aa3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bc598874cd78452bbfc3c21e26b35aa3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc598874cd78452bbfc3c21e26b35aa3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45861 } } }
I20260812 02:36:22.769629 5506 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc598874cd78452bbfc3c21e26b35aa3 [sys.catalog]: This master's current role is: LEADER
I20260812 02:36:22.770407 5510 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 02:36:22.771466 14748 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 02:36:22.773530 5510 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 02:36:22.779042 5510 catalog_manager.cc:1347] Generated new cluster ID: 39a23e079e60468381e2f92b3b81d4f4
I20260812 02:36:22.779109 5510 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 02:36:22.783707 5501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:22.773634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39171 (local address 127.0.0.1:45861)
0812 02:36:22.773740 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0812 02:36:22.773750 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 02:36:22.774071 (+ 321us) server_negotiation.cc:407] Connection header received
0812 02:36:22.774245 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:36:22.774249 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:36:22.774301 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:36:22.774388 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:36:22.775254 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:22.776029 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:22.777157 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:22.777504 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:22.777590 (+ 86us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:36:22.779618 (+ 2028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:36:22.779622 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:36:22.779625 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:36:22.779652 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:36:22.781383 (+ 1731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:36:22.782006 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:36:22.782010 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:36:22.782012 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:36:22.782109 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:36:22.782570 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:36:22.782573 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:36:22.782575 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:36:22.782832 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:36:22.782956 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0812 02:36:22.783431 (+ 475us) server_negotiation.cc:299] Negotiation successful
0812 02:36:22.783540 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260812 02:36:22.803774 5510 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 02:36:22.805135 5510 catalog_manager.cc:1504] Loading token signing keys...
I20260812 02:36:22.813905 5510 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bc598874cd78452bbfc3c21e26b35aa3: Generated new TSK 0
I20260812 02:36:22.814705 5510 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 02:36:22.822746 14748 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbep4M4/build/debug/bin/kudu with pid 5414
W20260812 02:36:22.830359 5394 connection.cc:535] client connection to 127.0.0.1:45861 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45861): Connection reset by peer (error 104)
2026-08-12T02:36:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1519 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786501898713699-14748-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 02:36:22 dist-test-slave-2w3w krb5kdc[5533](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 02:36:22 dist-test-slave-2w3w krb5kdc[5533](info): set up 2 sockets
Aug 12 02:36:22 dist-test-slave-2w3w krb5kdc[5533](info): commencing operation
krb5kdc: starting...
W20260812 02:36:23.966372 14748 mini_kdc.cc:121] Time spent starting KDC: real 1.121s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786501898713699-14748-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786501898713699-14748-0/krb5kdc/test-user.keytab.
Aug 12 02:36:24 dist-test-slave-2w3w krb5kdc[5533](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502184, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T02:36:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T02:36:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786501898713699-14748-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786501898713699-14748-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 02:36:24.110447 14748 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbep4M4/build/debug/bin/kudu
/tmp/dist-test-taskbep4M4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786501898713699-14748-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786501898713699-14748-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786501898713699-14748-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786501898713699-14748-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35131
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786501898713699-14748-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35131
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786501898713699-14748-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786501898713699-14748-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786501898713699-14748-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:36:24.223961 5547 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:36:24.224295 5547 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:36:24.224370 5547 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:36:24.228538 5547 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:36:24.228636 5547 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:36:24.228674 5547 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:36:24.228729 5547 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:36:24.228768 5547 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 02:36:24.234413 5547 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786501898713699-14748-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786501898713699-14748-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35131
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786501898713699-14748-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35131
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786501898713699-14748-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5547
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786501898713699-14748-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:36:24.235642 5547 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:36:24.236720 5547 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:36:24.245815 5557 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:36:24.245836 5554 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:36:24.245913 5553 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:36:24.246243 5547 server_base.cc:1034] running on GCE node
I20260812 02:36:24.246917 5547 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:36:24.248189 5547 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:36:24.249410 5547 hybrid_clock.cc:648] HybridClock initialized: now 1786502184249396 us; error 36 us; skew 500 ppm
Aug 12 02:36:24 dist-test-slave-2w3w krb5kdc[5533](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502184, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:36:24.252705 5547 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 02:36:24.253747 5547 webserver.cc:460] Webserver started at http://127.0.0.1:46777/ using document root <none> and password file <none>
I20260812 02:36:24.254382 5547 fs_manager.cc:359] Metadata directory not provided
I20260812 02:36:24.254487 5547 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:36:24.254756 5547 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:36:24.257223 5547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786501898713699-14748-0/minicluster-data/master-0/data/instance:
uuid: "400df0c31d264dfb882e8e8ae5f39d23"
format_stamp: "Formatted at 2026-08-12 02:36:24 on dist-test-slave-2w3w"
server_key: "7ca7f98a2ebaa8c57b7ad39c958f5c80"
server_key_iv: "87d92a4f42e33a9dc400a707c3285854"
server_key_version: "encryptionkey@0"
I20260812 02:36:24.257931 5547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786501898713699-14748-0/minicluster-data/master-0/wal/instance:
uuid: "400df0c31d264dfb882e8e8ae5f39d23"
format_stamp: "Formatted at 2026-08-12 02:36:24 on dist-test-slave-2w3w"
server_key: "7ca7f98a2ebaa8c57b7ad39c958f5c80"
server_key_iv: "87d92a4f42e33a9dc400a707c3285854"
server_key_version: "encryptionkey@0"
I20260812 02:36:24.262439 5547 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20260812 02:36:24.265537 5565 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:36:24.267130 5547 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 02:36:24.267290 5547 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786501898713699-14748-0/minicluster-data/master-0/data,/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786501898713699-14748-0/minicluster-data/master-0/wal
uuid: "400df0c31d264dfb882e8e8ae5f39d23"
format_stamp: "Formatted at 2026-08-12 02:36:24 on dist-test-slave-2w3w"
server_key: "7ca7f98a2ebaa8c57b7ad39c958f5c80"
server_key_iv: "87d92a4f42e33a9dc400a707c3285854"
server_key_version: "encryptionkey@0"
I20260812 02:36:24.267452 5547 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786501898713699-14748-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786501898713699-14748-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786501898713699-14748-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:36:24.280222 5547 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:36:24.283707 5547 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:36:24.283963 5547 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:36:24.291817 5547 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35131
I20260812 02:36:24.291846 5623 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35131 every 8 connection(s)
I20260812 02:36:24.292800 5547 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786501898713699-14748-0/minicluster-data/master-0/data/info.pb
I20260812 02:36:24.295549 5624 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:36:24.298110 14748 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbep4M4/build/debug/bin/kudu as pid 5547
I20260812 02:36:24.298257 14748 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786501898713699-14748-0/minicluster-data/master-0/wal/instance
I20260812 02:36:24.298559 14748 external_mini_cluster.cc:1428] Setting key 568dd3a0049082ef5150f9b6bfa576aa
I20260812 02:36:24.301510 5624 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 400df0c31d264dfb882e8e8ae5f39d23: Bootstrap starting.
I20260812 02:36:24.303714 5624 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 400df0c31d264dfb882e8e8ae5f39d23: Neither blocks nor log segments found. Creating new log.
I20260812 02:36:24.304462 5624 log.cc:824] T 00000000000000000000000000000000 P 400df0c31d264dfb882e8e8ae5f39d23: Log is configured to *not* fsync() on all Append() calls
I20260812 02:36:24.306505 5624 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 400df0c31d264dfb882e8e8ae5f39d23: No bootstrap required, opened a new log
I20260812 02:36:24.307860 5628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:24.300030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36353 (local address 127.0.0.1:35131)
0812 02:36:24.300502 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0812 02:36:24.300519 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0812 02:36:24.300556 (+ 37us) server_negotiation.cc:407] Connection header received
0812 02:36:24.301211 (+ 655us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:36:24.301228 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:36:24.301572 (+ 344us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:36:24.301940 (+ 368us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:36:24.302359 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:24.303420 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:24.304378 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:24.304901 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:24.305090 (+ 189us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:36:24.306067 (+ 977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:36:24.306072 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:36:24.306085 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:36:24.306503 (+ 418us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:36:24.306562 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:36:24.306584 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:36:24.306778 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0812 02:36:24.306951 (+ 173us) server_negotiation.cc:299] Negotiation successful
0812 02:36:24.307133 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":134,"threads_started":1}
I20260812 02:36:24.310323 5624 raft_consensus.cc:348] T 00000000000000000000000000000000 P 400df0c31d264dfb882e8e8ae5f39d23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "400df0c31d264dfb882e8e8ae5f39d23" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35131 } }
I20260812 02:36:24.310571 5624 raft_consensus.cc:374] T 00000000000000000000000000000000 P 400df0c31d264dfb882e8e8ae5f39d23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:36:24.310631 5624 raft_consensus.cc:729] T 00000000000000000000000000000000 P 400df0c31d264dfb882e8e8ae5f39d23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 400df0c31d264dfb882e8e8ae5f39d23, State: Initialized, Role: FOLLOWER
I20260812 02:36:24.310829 14748 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 02:36:24.311151 5624 consensus_queue.cc:260] T 00000000000000000000000000000000 P 400df0c31d264dfb882e8e8ae5f39d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "400df0c31d264dfb882e8e8ae5f39d23" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35131 } }
I20260812 02:36:24.311326 5624 raft_consensus.cc:388] T 00000000000000000000000000000000 P 400df0c31d264dfb882e8e8ae5f39d23 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 02:36:24.311400 5624 raft_consensus.cc:482] T 00000000000000000000000000000000 P 400df0c31d264dfb882e8e8ae5f39d23 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 02:36:24.311512 5624 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 400df0c31d264dfb882e8e8ae5f39d23 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:36:24.312813 5624 raft_consensus.cc:504] T 00000000000000000000000000000000 P 400df0c31d264dfb882e8e8ae5f39d23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "400df0c31d264dfb882e8e8ae5f39d23" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35131 } }
I20260812 02:36:24.313208 5624 leader_election.cc:302] T 00000000000000000000000000000000 P 400df0c31d264dfb882e8e8ae5f39d23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 400df0c31d264dfb882e8e8ae5f39d23; no voters:
I20260812 02:36:24.313594 5624 leader_election.cc:288] T 00000000000000000000000000000000 P 400df0c31d264dfb882e8e8ae5f39d23 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 02:36:24.313752 5635 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 400df0c31d264dfb882e8e8ae5f39d23 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:36:24.314014 5635 raft_consensus.cc:686] T 00000000000000000000000000000000 P 400df0c31d264dfb882e8e8ae5f39d23 [term 1 LEADER]: Becoming Leader. State: Replica: 400df0c31d264dfb882e8e8ae5f39d23, State: Running, Role: LEADER
I20260812 02:36:24.314411 5635 consensus_queue.cc:237] T 00000000000000000000000000000000 P 400df0c31d264dfb882e8e8ae5f39d23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "400df0c31d264dfb882e8e8ae5f39d23" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35131 } }
I20260812 02:36:24.315001 5624 sys_catalog.cc:564] T 00000000000000000000000000000000 P 400df0c31d264dfb882e8e8ae5f39d23 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:36:24.316097 5636 sys_catalog.cc:455] T 00000000000000000000000000000000 P 400df0c31d264dfb882e8e8ae5f39d23 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "400df0c31d264dfb882e8e8ae5f39d23" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "400df0c31d264dfb882e8e8ae5f39d23" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35131 } } }
I20260812 02:36:24.316226 5636 sys_catalog.cc:458] T 00000000000000000000000000000000 P 400df0c31d264dfb882e8e8ae5f39d23 [sys.catalog]: This master's current role is: LEADER
I20260812 02:36:24.316576 5637 sys_catalog.cc:455] T 00000000000000000000000000000000 P 400df0c31d264dfb882e8e8ae5f39d23 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 400df0c31d264dfb882e8e8ae5f39d23. Latest consensus state: current_term: 1 leader_uuid: "400df0c31d264dfb882e8e8ae5f39d23" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "400df0c31d264dfb882e8e8ae5f39d23" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35131 } } }
I20260812 02:36:24.316658 5637 sys_catalog.cc:458] T 00000000000000000000000000000000 P 400df0c31d264dfb882e8e8ae5f39d23 [sys.catalog]: This master's current role is: LEADER
I20260812 02:36:24.318760 5628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:24.313253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35851 (local address 127.0.0.1:35131)
0812 02:36:24.313394 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0812 02:36:24.313401 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 02:36:24.313645 (+ 244us) server_negotiation.cc:407] Connection header received
0812 02:36:24.313782 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:36:24.313786 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:36:24.313842 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:36:24.313923 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:36:24.314353 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:24.315132 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:24.316372 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:24.316749 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:24.316829 (+ 80us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:36:24.318254 (+ 1425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:36:24.318258 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:36:24.318260 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:36:24.318273 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:36:24.318305 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:36:24.318316 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:36:24.318419 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0812 02:36:24.318545 (+ 126us) server_negotiation.cc:299] Negotiation successful
0812 02:36:24.318579 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
W20260812 02:36:24.320559 5652 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 400df0c31d264dfb882e8e8ae5f39d23: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 02:36:24.320645 5652 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 02:36:24.320753 5645 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 02:36:24.321548 5645 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 02:36:24.327378 5645 catalog_manager.cc:1347] Generated new cluster ID: 6106a57075194717be723da59c7b4dd7
I20260812 02:36:24.327471 5645 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 02:36:24.352886 5645 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 02:36:24.354235 5645 catalog_manager.cc:1504] Loading token signing keys...
I20260812 02:36:24.367660 5645 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 400df0c31d264dfb882e8e8ae5f39d23: Generated new TSK 0
I20260812 02:36:24.368512 5645 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 02:36:24.377561 14748 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbep4M4/build/debug/bin/kudu with pid 5547
W20260812 02:36:24.386829 5530 connection.cc:535] client connection to 127.0.0.1:35131 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35131): Connection reset by peer (error 104)
2026-08-12T02:36:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1556 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786501898713699-14748-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 02:36:24 dist-test-slave-2w3w krb5kdc[5659](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 02:36:24 dist-test-slave-2w3w krb5kdc[5659](info): set up 2 sockets
Aug 12 02:36:24 dist-test-slave-2w3w krb5kdc[5659](info): commencing operation
krb5kdc: starting...
W20260812 02:36:25.520808 14748 mini_kdc.cc:121] Time spent starting KDC: real 1.119s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786501898713699-14748-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786501898713699-14748-0/krb5kdc/test-user.keytab.
Aug 12 02:36:25 dist-test-slave-2w3w krb5kdc[5659](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502185, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T02:36:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T02:36:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786501898713699-14748-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786501898713699-14748-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 02:36:25.662865 14748 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbep4M4/build/debug/bin/kudu
/tmp/dist-test-taskbep4M4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786501898713699-14748-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786501898713699-14748-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786501898713699-14748-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786501898713699-14748-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37171
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786501898713699-14748-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37171
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786501898713699-14748-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786501898713699-14748-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786501898713699-14748-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:36:25.776153 5674 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:36:25.776427 5674 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:36:25.776472 5674 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:36:25.780465 5674 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:36:25.780530 5674 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:36:25.780550 5674 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:36:25.780567 5674 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:36:25.780582 5674 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 02:36:25.785863 5674 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786501898713699-14748-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786501898713699-14748-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37171
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786501898713699-14748-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37171
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786501898713699-14748-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5674
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786501898713699-14748-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:36:25.786932 5674 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:36:25.787992 5674 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:36:25.796752 5680 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:36:25.796765 5686 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:36:25.797297 5683 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:36:25.797442 5674 server_base.cc:1034] running on GCE node
I20260812 02:36:25.798044 5674 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:36:25.799254 5674 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:36:25.800400 5674 hybrid_clock.cc:648] HybridClock initialized: now 1786502185800381 us; error 33 us; skew 500 ppm
Aug 12 02:36:25 dist-test-slave-2w3w krb5kdc[5659](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502185, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:36:25.807207 5674 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 02:36:25.808238 5674 webserver.cc:460] Webserver started at http://127.0.0.1:35649/ using document root <none> and password file <none>
I20260812 02:36:25.808804 5674 fs_manager.cc:359] Metadata directory not provided
I20260812 02:36:25.808877 5674 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:36:25.809059 5674 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:36:25.811321 5674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786501898713699-14748-0/minicluster-data/master-0/data/instance:
uuid: "df2eee4a5ed04ac0af48d22f20cd62bb"
format_stamp: "Formatted at 2026-08-12 02:36:25 on dist-test-slave-2w3w"
server_key: "adb77ac084444f5d3ce2e179c07ebd8a"
server_key_iv: "580eab597c8b950f75aab31169a891be"
server_key_version: "encryptionkey@0"
I20260812 02:36:25.811884 5674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786501898713699-14748-0/minicluster-data/master-0/wal/instance:
uuid: "df2eee4a5ed04ac0af48d22f20cd62bb"
format_stamp: "Formatted at 2026-08-12 02:36:25 on dist-test-slave-2w3w"
server_key: "adb77ac084444f5d3ce2e179c07ebd8a"
server_key_iv: "580eab597c8b950f75aab31169a891be"
server_key_version: "encryptionkey@0"
I20260812 02:36:25.815734 5674 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260812 02:36:25.818264 5693 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:36:25.819680 5674 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260812 02:36:25.819805 5674 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786501898713699-14748-0/minicluster-data/master-0/data,/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786501898713699-14748-0/minicluster-data/master-0/wal
uuid: "df2eee4a5ed04ac0af48d22f20cd62bb"
format_stamp: "Formatted at 2026-08-12 02:36:25 on dist-test-slave-2w3w"
server_key: "adb77ac084444f5d3ce2e179c07ebd8a"
server_key_iv: "580eab597c8b950f75aab31169a891be"
server_key_version: "encryptionkey@0"
I20260812 02:36:25.819924 5674 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786501898713699-14748-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786501898713699-14748-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786501898713699-14748-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:36:25.825098 5674 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:36:25.828430 5674 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:36:25.828636 5674 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:36:25.836066 5674 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37171
I20260812 02:36:25.836088 5755 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37171 every 8 connection(s)
I20260812 02:36:25.837044 5674 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786501898713699-14748-0/minicluster-data/master-0/data/info.pb
I20260812 02:36:25.839918 5756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:36:25.840690 14748 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbep4M4/build/debug/bin/kudu as pid 5674
I20260812 02:36:25.840859 14748 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786501898713699-14748-0/minicluster-data/master-0/wal/instance
I20260812 02:36:25.841229 14748 external_mini_cluster.cc:1428] Setting key 879d50eaae6e657716c8cb53ea5497a0
I20260812 02:36:25.847129 5760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:25.842820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39255 (local address 127.0.0.1:37171)
0812 02:36:25.843353 (+ 533us) server_negotiation.cc:206] Beginning negotiation
0812 02:36:25.843372 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0812 02:36:25.843413 (+ 41us) server_negotiation.cc:407] Connection header received
0812 02:36:25.844310 (+ 897us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:36:25.844330 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:36:25.844633 (+ 303us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:36:25.845023 (+ 390us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:36:25.845850 (+ 827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:36:25.845855 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:36:25.845866 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:36:25.845877 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:36:25.845939 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:36:25.845959 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:36:25.846114 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0812 02:36:25.846187 (+ 73us) server_negotiation.cc:299] Negotiation successful
0812 02:36:25.846346 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":142,"threads_started":1}
I20260812 02:36:25.847159 5756 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df2eee4a5ed04ac0af48d22f20cd62bb: Bootstrap starting.
I20260812 02:36:25.850142 5756 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P df2eee4a5ed04ac0af48d22f20cd62bb: Neither blocks nor log segments found. Creating new log.
I20260812 02:36:25.850507 14748 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 02:36:25.851181 5756 log.cc:824] T 00000000000000000000000000000000 P df2eee4a5ed04ac0af48d22f20cd62bb: Log is configured to *not* fsync() on all Append() calls
I20260812 02:36:25.853930 5756 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df2eee4a5ed04ac0af48d22f20cd62bb: No bootstrap required, opened a new log
I20260812 02:36:25.854820 5760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:25.852851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50169 (local address 127.0.0.1:37171)
0812 02:36:25.852998 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0812 02:36:25.853002 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:36:25.853245 (+ 243us) server_negotiation.cc:407] Connection header received
0812 02:36:25.853343 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:36:25.853346 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:36:25.853398 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:36:25.853478 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:36:25.854326 (+ 848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:36:25.854329 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:36:25.854330 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:36:25.854342 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:36:25.854371 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:36:25.854382 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:36:25.854470 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0812 02:36:25.854595 (+ 125us) server_negotiation.cc:299] Negotiation successful
0812 02:36:25.854635 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260812 02:36:25.858744 5756 raft_consensus.cc:348] T 00000000000000000000000000000000 P df2eee4a5ed04ac0af48d22f20cd62bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df2eee4a5ed04ac0af48d22f20cd62bb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37171 } }
I20260812 02:36:25.859158 5756 raft_consensus.cc:374] T 00000000000000000000000000000000 P df2eee4a5ed04ac0af48d22f20cd62bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:36:25.859238 5756 raft_consensus.cc:729] T 00000000000000000000000000000000 P df2eee4a5ed04ac0af48d22f20cd62bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df2eee4a5ed04ac0af48d22f20cd62bb, State: Initialized, Role: FOLLOWER
I20260812 02:36:25.859722 5756 consensus_queue.cc:260] T 00000000000000000000000000000000 P df2eee4a5ed04ac0af48d22f20cd62bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df2eee4a5ed04ac0af48d22f20cd62bb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37171 } }
I20260812 02:36:25.859869 5756 raft_consensus.cc:388] T 00000000000000000000000000000000 P df2eee4a5ed04ac0af48d22f20cd62bb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 02:36:25.859916 5756 raft_consensus.cc:482] T 00000000000000000000000000000000 P df2eee4a5ed04ac0af48d22f20cd62bb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 02:36:25.859990 5756 raft_consensus.cc:3037] T 00000000000000000000000000000000 P df2eee4a5ed04ac0af48d22f20cd62bb [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:36:25.861239 5756 raft_consensus.cc:504] T 00000000000000000000000000000000 P df2eee4a5ed04ac0af48d22f20cd62bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df2eee4a5ed04ac0af48d22f20cd62bb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37171 } }
I20260812 02:36:25.861651 5756 leader_election.cc:302] T 00000000000000000000000000000000 P df2eee4a5ed04ac0af48d22f20cd62bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: df2eee4a5ed04ac0af48d22f20cd62bb; no voters:
I20260812 02:36:25.862037 5756 leader_election.cc:288] T 00000000000000000000000000000000 P df2eee4a5ed04ac0af48d22f20cd62bb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 02:36:25.862212 5771 raft_consensus.cc:2781] T 00000000000000000000000000000000 P df2eee4a5ed04ac0af48d22f20cd62bb [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:36:25.862485 5771 raft_consensus.cc:686] T 00000000000000000000000000000000 P df2eee4a5ed04ac0af48d22f20cd62bb [term 1 LEADER]: Becoming Leader. State: Replica: df2eee4a5ed04ac0af48d22f20cd62bb, State: Running, Role: LEADER
I20260812 02:36:25.862854 5771 consensus_queue.cc:237] T 00000000000000000000000000000000 P df2eee4a5ed04ac0af48d22f20cd62bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df2eee4a5ed04ac0af48d22f20cd62bb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37171 } }
I20260812 02:36:25.863507 5756 sys_catalog.cc:564] T 00000000000000000000000000000000 P df2eee4a5ed04ac0af48d22f20cd62bb [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:36:25.865113 5772 sys_catalog.cc:455] T 00000000000000000000000000000000 P df2eee4a5ed04ac0af48d22f20cd62bb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "df2eee4a5ed04ac0af48d22f20cd62bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df2eee4a5ed04ac0af48d22f20cd62bb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37171 } } }
I20260812 02:36:25.865284 5772 sys_catalog.cc:458] T 00000000000000000000000000000000 P df2eee4a5ed04ac0af48d22f20cd62bb [sys.catalog]: This master's current role is: LEADER
I20260812 02:36:25.866024 5782 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 02:36:25.866272 5773 sys_catalog.cc:455] T 00000000000000000000000000000000 P df2eee4a5ed04ac0af48d22f20cd62bb [sys.catalog]: SysCatalogTable state changed. Reason: New leader df2eee4a5ed04ac0af48d22f20cd62bb. Latest consensus state: current_term: 1 leader_uuid: "df2eee4a5ed04ac0af48d22f20cd62bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df2eee4a5ed04ac0af48d22f20cd62bb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37171 } } }
I20260812 02:36:25.866771 5773 sys_catalog.cc:458] T 00000000000000000000000000000000 P df2eee4a5ed04ac0af48d22f20cd62bb [sys.catalog]: This master's current role is: LEADER
I20260812 02:36:25.869257 5782 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 02:36:25.876425 5782 catalog_manager.cc:1347] Generated new cluster ID: 569525b5858949759171f84a158a566f
I20260812 02:36:25.876544 5782 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 02:36:25.890776 5782 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 02:36:25.891300 5782 catalog_manager.cc:1504] Loading token signing keys...
I20260812 02:36:25.904413 5782 catalog_manager.cc:5868] T 00000000000000000000000000000000 P df2eee4a5ed04ac0af48d22f20cd62bb: Generated new TSK 0
I20260812 02:36:25.905438 5782 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 02:36:25.915134 14748 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbep4M4/build/debug/bin/kudu with pid 5674
2026-08-12T02:36:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1536 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786501898713699-14748-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 02:36:25 dist-test-slave-2w3w krb5kdc[5795](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 02:36:25 dist-test-slave-2w3w krb5kdc[5795](info): set up 2 sockets
Aug 12 02:36:25 dist-test-slave-2w3w krb5kdc[5795](info): commencing operation
krb5kdc: starting...
W20260812 02:36:27.033387 14748 mini_kdc.cc:121] Time spent starting KDC: real 1.095s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786501898713699-14748-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786501898713699-14748-0/krb5kdc/test-user.keytab.
Aug 12 02:36:27 dist-test-slave-2w3w krb5kdc[5795](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502187, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T02:36:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T02:36:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786501898713699-14748-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786501898713699-14748-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 02:36:27.179579 14748 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbep4M4/build/debug/bin/kudu
/tmp/dist-test-taskbep4M4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786501898713699-14748-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786501898713699-14748-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786501898713699-14748-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786501898713699-14748-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42799
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786501898713699-14748-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42799
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786501898713699-14748-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786501898713699-14748-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786501898713699-14748-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:36:27.292419 5809 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:36:27.292754 5809 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:36:27.292819 5809 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:36:27.297063 5809 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:36:27.297166 5809 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:36:27.297206 5809 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:36:27.297261 5809 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:36:27.297322 5809 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 02:36:27.302906 5809 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786501898713699-14748-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786501898713699-14748-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42799
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786501898713699-14748-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42799
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786501898713699-14748-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5809
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786501898713699-14748-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:36:27.304098 5809 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:36:27.305176 5809 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:36:27.314105 5815 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:36:27.314206 5814 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:36:27.314105 5817 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:36:27.314327 5809 server_base.cc:1034] running on GCE node
I20260812 02:36:27.315152 5809 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:36:27.316342 5809 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:36:27.317561 5809 hybrid_clock.cc:648] HybridClock initialized: now 1786502187317530 us; error 44 us; skew 500 ppm
Aug 12 02:36:27 dist-test-slave-2w3w krb5kdc[5795](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502187, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:36:27.320745 5809 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 02:36:27.321699 5809 webserver.cc:460] Webserver started at http://127.0.0.1:46085/ using document root <none> and password file <none>
I20260812 02:36:27.322295 5809 fs_manager.cc:359] Metadata directory not provided
I20260812 02:36:27.322389 5809 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:36:27.322625 5809 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:36:27.324863 5809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786501898713699-14748-0/minicluster-data/master-0/data/instance:
uuid: "c15aaaedc4bf4df89755dd08acd5ffee"
format_stamp: "Formatted at 2026-08-12 02:36:27 on dist-test-slave-2w3w"
server_key: "7f5bfb7725ac4f5fa062ac6c4b89f899"
server_key_iv: "8f9e20ff2fd9a745159a198a533f2743"
server_key_version: "encryptionkey@0"
I20260812 02:36:27.325752 5809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786501898713699-14748-0/minicluster-data/master-0/wal/instance:
uuid: "c15aaaedc4bf4df89755dd08acd5ffee"
format_stamp: "Formatted at 2026-08-12 02:36:27 on dist-test-slave-2w3w"
server_key: "7f5bfb7725ac4f5fa062ac6c4b89f899"
server_key_iv: "8f9e20ff2fd9a745159a198a533f2743"
server_key_version: "encryptionkey@0"
I20260812 02:36:27.329882 5809 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260812 02:36:27.332465 5824 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:36:27.333696 5809 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260812 02:36:27.333842 5809 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786501898713699-14748-0/minicluster-data/master-0/data,/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786501898713699-14748-0/minicluster-data/master-0/wal
uuid: "c15aaaedc4bf4df89755dd08acd5ffee"
format_stamp: "Formatted at 2026-08-12 02:36:27 on dist-test-slave-2w3w"
server_key: "7f5bfb7725ac4f5fa062ac6c4b89f899"
server_key_iv: "8f9e20ff2fd9a745159a198a533f2743"
server_key_version: "encryptionkey@0"
I20260812 02:36:27.333957 5809 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786501898713699-14748-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786501898713699-14748-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786501898713699-14748-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:36:27.347200 5809 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:36:27.350142 5809 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:36:27.350348 5809 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:36:27.357728 5809 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42799
I20260812 02:36:27.357751 5885 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42799 every 8 connection(s)
I20260812 02:36:27.358659 5809 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786501898713699-14748-0/minicluster-data/master-0/data/info.pb
I20260812 02:36:27.361383 5886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:36:27.366896 5886 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c15aaaedc4bf4df89755dd08acd5ffee: Bootstrap starting.
I20260812 02:36:27.367391 14748 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbep4M4/build/debug/bin/kudu as pid 5809
I20260812 02:36:27.367560 14748 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786501898713699-14748-0/minicluster-data/master-0/wal/instance
I20260812 02:36:27.367852 14748 external_mini_cluster.cc:1428] Setting key 5571d15d0f8665758a48864661a3d2b3
I20260812 02:36:27.369663 5886 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c15aaaedc4bf4df89755dd08acd5ffee: Neither blocks nor log segments found. Creating new log.
I20260812 02:36:27.370406 5886 log.cc:824] T 00000000000000000000000000000000 P c15aaaedc4bf4df89755dd08acd5ffee: Log is configured to *not* fsync() on all Append() calls
I20260812 02:36:27.372722 5886 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c15aaaedc4bf4df89755dd08acd5ffee: No bootstrap required, opened a new log
Aug 12 02:36:27 dist-test-slave-2w3w krb5kdc[5795](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502187, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260812 02:36:27.377591 5886 raft_consensus.cc:348] T 00000000000000000000000000000000 P c15aaaedc4bf4df89755dd08acd5ffee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c15aaaedc4bf4df89755dd08acd5ffee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42799 } }
I20260812 02:36:27.377894 5886 raft_consensus.cc:374] T 00000000000000000000000000000000 P c15aaaedc4bf4df89755dd08acd5ffee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:36:27.377969 5886 raft_consensus.cc:729] T 00000000000000000000000000000000 P c15aaaedc4bf4df89755dd08acd5ffee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c15aaaedc4bf4df89755dd08acd5ffee, State: Initialized, Role: FOLLOWER
I20260812 02:36:27.378474 5886 consensus_queue.cc:260] T 00000000000000000000000000000000 P c15aaaedc4bf4df89755dd08acd5ffee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c15aaaedc4bf4df89755dd08acd5ffee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42799 } }
I20260812 02:36:27.378656 5886 raft_consensus.cc:388] T 00000000000000000000000000000000 P c15aaaedc4bf4df89755dd08acd5ffee [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 02:36:27.378736 5886 raft_consensus.cc:482] T 00000000000000000000000000000000 P c15aaaedc4bf4df89755dd08acd5ffee [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 02:36:27.378852 5886 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c15aaaedc4bf4df89755dd08acd5ffee [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:36:27.380192 5886 raft_consensus.cc:504] T 00000000000000000000000000000000 P c15aaaedc4bf4df89755dd08acd5ffee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c15aaaedc4bf4df89755dd08acd5ffee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42799 } }
I20260812 02:36:27.380578 5886 leader_election.cc:302] T 00000000000000000000000000000000 P c15aaaedc4bf4df89755dd08acd5ffee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c15aaaedc4bf4df89755dd08acd5ffee; no voters:
I20260812 02:36:27.380930 5886 leader_election.cc:288] T 00000000000000000000000000000000 P c15aaaedc4bf4df89755dd08acd5ffee [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 02:36:27.381062 5891 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c15aaaedc4bf4df89755dd08acd5ffee [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:36:27.381348 5891 raft_consensus.cc:686] T 00000000000000000000000000000000 P c15aaaedc4bf4df89755dd08acd5ffee [term 1 LEADER]: Becoming Leader. State: Replica: c15aaaedc4bf4df89755dd08acd5ffee, State: Running, Role: LEADER
I20260812 02:36:27.381692 5891 consensus_queue.cc:237] T 00000000000000000000000000000000 P c15aaaedc4bf4df89755dd08acd5ffee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c15aaaedc4bf4df89755dd08acd5ffee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42799 } }
I20260812 02:36:27.382359 5886 sys_catalog.cc:564] T 00000000000000000000000000000000 P c15aaaedc4bf4df89755dd08acd5ffee [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:36:27.383379 5892 sys_catalog.cc:455] T 00000000000000000000000000000000 P c15aaaedc4bf4df89755dd08acd5ffee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c15aaaedc4bf4df89755dd08acd5ffee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c15aaaedc4bf4df89755dd08acd5ffee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42799 } } }
I20260812 02:36:27.383517 5892 sys_catalog.cc:458] T 00000000000000000000000000000000 P c15aaaedc4bf4df89755dd08acd5ffee [sys.catalog]: This master's current role is: LEADER
I20260812 02:36:27.383917 5893 sys_catalog.cc:455] T 00000000000000000000000000000000 P c15aaaedc4bf4df89755dd08acd5ffee [sys.catalog]: SysCatalogTable state changed. Reason: New leader c15aaaedc4bf4df89755dd08acd5ffee. Latest consensus state: current_term: 1 leader_uuid: "c15aaaedc4bf4df89755dd08acd5ffee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c15aaaedc4bf4df89755dd08acd5ffee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42799 } } }
I20260812 02:36:27.384056 5893 sys_catalog.cc:458] T 00000000000000000000000000000000 P c15aaaedc4bf4df89755dd08acd5ffee [sys.catalog]: This master's current role is: LEADER
I20260812 02:36:27.384701 5905 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 02:36:27.384861 5889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:27.369205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35689 (local address 127.0.0.1:42799)
0812 02:36:27.369642 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0812 02:36:27.369658 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0812 02:36:27.369695 (+ 37us) server_negotiation.cc:407] Connection header received
0812 02:36:27.370446 (+ 751us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:36:27.370464 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:36:27.370918 (+ 454us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:36:27.371360 (+ 442us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:36:27.372302 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:27.373382 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:27.374566 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:27.375181 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:27.375307 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:36:27.378406 (+ 3099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:36:27.378412 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:36:27.378428 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:36:27.378462 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:36:27.380604 (+ 2142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:36:27.381305 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:36:27.381312 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:36:27.381318 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:36:27.381488 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:36:27.381822 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:36:27.381825 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:36:27.381826 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:36:27.382355 (+ 529us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:36:27.382560 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0812 02:36:27.383434 (+ 874us) server_negotiation.cc:299] Negotiation successful
0812 02:36:27.383799 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":142,"threads_started":1}
I20260812 02:36:27.388017 14748 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 02:36:27.388233 5905 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 02:36:27.394531 5905 catalog_manager.cc:1347] Generated new cluster ID: 799d1b6179734140b26b35c44849cab4
I20260812 02:36:27.395527 5905 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 02:36:27.399274 5889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:27.390101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42245 (local address 127.0.0.1:42799)
0812 02:36:27.390222 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0812 02:36:27.390226 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:36:27.390407 (+ 181us) server_negotiation.cc:407] Connection header received
0812 02:36:27.390536 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:36:27.390539 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:36:27.390592 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:36:27.390685 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:36:27.391611 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:27.392437 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:27.393482 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:27.393865 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:27.393978 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:36:27.395554 (+ 1576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:36:27.395558 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:36:27.395561 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:36:27.395588 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:36:27.397477 (+ 1889us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:36:27.397967 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:36:27.397971 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:36:27.397973 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:36:27.398071 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:36:27.398342 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:36:27.398346 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:36:27.398348 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:36:27.398575 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:36:27.398712 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0812 02:36:27.398947 (+ 235us) server_negotiation.cc:299] Negotiation successful
0812 02:36:27.399111 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260812 02:36:27.409436 5905 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 02:36:27.410773 5905 catalog_manager.cc:1504] Loading token signing keys...
I20260812 02:36:27.421345 5905 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c15aaaedc4bf4df89755dd08acd5ffee: Generated new TSK 0
I20260812 02:36:27.422281 5905 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 02:36:27.434240 14748 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbep4M4/build/debug/bin/kudu with pid 5809
2026-08-12T02:36:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1521 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786501898713699-14748-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 02:36:27 dist-test-slave-2w3w krb5kdc[5929](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 02:36:27 dist-test-slave-2w3w krb5kdc[5929](info): set up 2 sockets
Aug 12 02:36:27 dist-test-slave-2w3w krb5kdc[5929](info): commencing operation
krb5kdc: starting...
W20260812 02:36:28.596801 14748 mini_kdc.cc:121] Time spent starting KDC: real 1.136s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786501898713699-14748-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786501898713699-14748-0/krb5kdc/test-user.keytab.
Aug 12 02:36:28 dist-test-slave-2w3w krb5kdc[5929](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502188, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T02:36:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T02:36:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786501898713699-14748-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786501898713699-14748-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 02:36:28.742821 14748 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbep4M4/build/debug/bin/kudu
/tmp/dist-test-taskbep4M4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786501898713699-14748-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786501898713699-14748-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786501898713699-14748-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786501898713699-14748-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44091
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786501898713699-14748-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44091
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786501898713699-14748-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786501898713699-14748-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786501898713699-14748-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:36:28.859498 5943 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:36:28.859871 5943 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:36:28.859968 5943 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:36:28.864132 5943 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:36:28.864253 5943 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:36:28.864296 5943 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:36:28.864336 5943 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:36:28.864370 5943 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 02:36:28.870000 5943 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786501898713699-14748-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786501898713699-14748-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44091
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786501898713699-14748-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44091
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786501898713699-14748-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5943
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786501898713699-14748-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:36:28.871244 5943 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:36:28.872321 5943 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:36:28.881240 5948 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:36:28.881258 5951 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:36:28.881263 5949 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:36:28.881302 5943 server_base.cc:1034] running on GCE node
I20260812 02:36:28.882123 5943 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:36:28.883450 5943 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:36:28.884670 5943 hybrid_clock.cc:648] HybridClock initialized: now 1786502188884640 us; error 49 us; skew 500 ppm
Aug 12 02:36:28 dist-test-slave-2w3w krb5kdc[5929](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502188, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:36:28.888000 5943 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 02:36:28.889125 5943 webserver.cc:460] Webserver started at http://127.0.0.1:40921/ using document root <none> and password file <none>
I20260812 02:36:28.889741 5943 fs_manager.cc:359] Metadata directory not provided
I20260812 02:36:28.889851 5943 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:36:28.890105 5943 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:36:28.892498 5943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786501898713699-14748-0/minicluster-data/master-0/data/instance:
uuid: "0c7c0dfd3fdd4ae0a14f6e96ce847526"
format_stamp: "Formatted at 2026-08-12 02:36:28 on dist-test-slave-2w3w"
server_key: "abfe1f9b93e27a4f99d180361d40a18d"
server_key_iv: "150facda8788de8672a3c34cde8f6cbd"
server_key_version: "encryptionkey@0"
I20260812 02:36:28.893122 5943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786501898713699-14748-0/minicluster-data/master-0/wal/instance:
uuid: "0c7c0dfd3fdd4ae0a14f6e96ce847526"
format_stamp: "Formatted at 2026-08-12 02:36:28 on dist-test-slave-2w3w"
server_key: "abfe1f9b93e27a4f99d180361d40a18d"
server_key_iv: "150facda8788de8672a3c34cde8f6cbd"
server_key_version: "encryptionkey@0"
I20260812 02:36:28.897424 5943 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260812 02:36:28.900244 5959 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:36:28.901618 5943 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 02:36:28.901799 5943 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786501898713699-14748-0/minicluster-data/master-0/data,/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786501898713699-14748-0/minicluster-data/master-0/wal
uuid: "0c7c0dfd3fdd4ae0a14f6e96ce847526"
format_stamp: "Formatted at 2026-08-12 02:36:28 on dist-test-slave-2w3w"
server_key: "abfe1f9b93e27a4f99d180361d40a18d"
server_key_iv: "150facda8788de8672a3c34cde8f6cbd"
server_key_version: "encryptionkey@0"
I20260812 02:36:28.901940 5943 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786501898713699-14748-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786501898713699-14748-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786501898713699-14748-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:36:28.914937 5943 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:36:28.918473 5943 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:36:28.918715 5943 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:36:28.926829 5943 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44091
I20260812 02:36:28.926851 6016 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44091 every 8 connection(s)
I20260812 02:36:28.927966 5943 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786501898713699-14748-0/minicluster-data/master-0/data/info.pb
I20260812 02:36:28.931051 14748 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbep4M4/build/debug/bin/kudu as pid 5943
I20260812 02:36:28.931188 6017 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:36:28.931222 14748 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786501898713699-14748-0/minicluster-data/master-0/wal/instance
I20260812 02:36:28.931541 14748 external_mini_cluster.cc:1428] Setting key 81d435b1b9c85065b3fbaa1c376a8ba7
I20260812 02:36:28.938270 6017 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c7c0dfd3fdd4ae0a14f6e96ce847526: Bootstrap starting.
I20260812 02:36:28.940871 6017 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0c7c0dfd3fdd4ae0a14f6e96ce847526: Neither blocks nor log segments found. Creating new log.
I20260812 02:36:28.941884 6017 log.cc:824] T 00000000000000000000000000000000 P 0c7c0dfd3fdd4ae0a14f6e96ce847526: Log is configured to *not* fsync() on all Append() calls
I20260812 02:36:28.942071 6020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:28.933329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57595 (local address 127.0.0.1:44091)
0812 02:36:28.933885 (+ 556us) server_negotiation.cc:206] Beginning negotiation
0812 02:36:28.933902 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0812 02:36:28.933942 (+ 40us) server_negotiation.cc:407] Connection header received
0812 02:36:28.934697 (+ 755us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:36:28.934718 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:36:28.935236 (+ 518us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:36:28.935623 (+ 387us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:36:28.936063 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:28.937112 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:28.938243 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:28.938828 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:28.939038 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:36:28.940017 (+ 979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:36:28.940022 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:36:28.940036 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:36:28.940593 (+ 557us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:36:28.940665 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:36:28.940691 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:36:28.940830 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0812 02:36:28.941029 (+ 199us) server_negotiation.cc:299] Negotiation successful
0812 02:36:28.941278 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":383,"thread_start_us":166,"threads_started":1}
I20260812 02:36:28.944335 6017 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c7c0dfd3fdd4ae0a14f6e96ce847526: No bootstrap required, opened a new log
I20260812 02:36:28.945371 14748 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 02:36:28.948709 6017 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0c7c0dfd3fdd4ae0a14f6e96ce847526 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c7c0dfd3fdd4ae0a14f6e96ce847526" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44091 } }
I20260812 02:36:28.949024 6017 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0c7c0dfd3fdd4ae0a14f6e96ce847526 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:36:28.949095 6017 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0c7c0dfd3fdd4ae0a14f6e96ce847526 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c7c0dfd3fdd4ae0a14f6e96ce847526, State: Initialized, Role: FOLLOWER
I20260812 02:36:28.949630 6017 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0c7c0dfd3fdd4ae0a14f6e96ce847526 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c7c0dfd3fdd4ae0a14f6e96ce847526" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44091 } }
I20260812 02:36:28.949823 6017 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0c7c0dfd3fdd4ae0a14f6e96ce847526 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 02:36:28.949887 6017 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0c7c0dfd3fdd4ae0a14f6e96ce847526 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 02:36:28.949990 6017 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0c7c0dfd3fdd4ae0a14f6e96ce847526 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:36:28.951316 6017 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0c7c0dfd3fdd4ae0a14f6e96ce847526 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c7c0dfd3fdd4ae0a14f6e96ce847526" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44091 } }
I20260812 02:36:28.951779 6017 leader_election.cc:302] T 00000000000000000000000000000000 P 0c7c0dfd3fdd4ae0a14f6e96ce847526 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0c7c0dfd3fdd4ae0a14f6e96ce847526; no voters:
I20260812 02:36:28.952231 6017 leader_election.cc:288] T 00000000000000000000000000000000 P 0c7c0dfd3fdd4ae0a14f6e96ce847526 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 02:36:28.952376 6029 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0c7c0dfd3fdd4ae0a14f6e96ce847526 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:36:28.952556 6020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:28.947622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38313 (local address 127.0.0.1:44091)
0812 02:36:28.947781 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0812 02:36:28.947786 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:36:28.948019 (+ 233us) server_negotiation.cc:407] Connection header received
0812 02:36:28.948188 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:36:28.948192 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:36:28.948244 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:36:28.948330 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:36:28.948618 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:28.949276 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:28.950503 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:28.950819 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:28.950926 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:36:28.951964 (+ 1038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:36:28.951969 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:36:28.951972 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:36:28.951989 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:36:28.952024 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:36:28.952039 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:36:28.952130 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0812 02:36:28.952323 (+ 193us) server_negotiation.cc:299] Negotiation successful
0812 02:36:28.952405 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260812 02:36:28.953074 6029 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0c7c0dfd3fdd4ae0a14f6e96ce847526 [term 1 LEADER]: Becoming Leader. State: Replica: 0c7c0dfd3fdd4ae0a14f6e96ce847526, State: Running, Role: LEADER
I20260812 02:36:28.953465 6029 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0c7c0dfd3fdd4ae0a14f6e96ce847526 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c7c0dfd3fdd4ae0a14f6e96ce847526" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44091 } }
I20260812 02:36:28.954082 6017 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0c7c0dfd3fdd4ae0a14f6e96ce847526 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:36:28.954804 6029 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c7c0dfd3fdd4ae0a14f6e96ce847526 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0c7c0dfd3fdd4ae0a14f6e96ce847526. Latest consensus state: current_term: 1 leader_uuid: "0c7c0dfd3fdd4ae0a14f6e96ce847526" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c7c0dfd3fdd4ae0a14f6e96ce847526" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44091 } } }
I20260812 02:36:28.954941 6029 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c7c0dfd3fdd4ae0a14f6e96ce847526 [sys.catalog]: This master's current role is: LEADER
I20260812 02:36:28.955567 6031 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c7c0dfd3fdd4ae0a14f6e96ce847526 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0c7c0dfd3fdd4ae0a14f6e96ce847526" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c7c0dfd3fdd4ae0a14f6e96ce847526" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44091 } } }
I20260812 02:36:28.955679 6031 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c7c0dfd3fdd4ae0a14f6e96ce847526 [sys.catalog]: This master's current role is: LEADER
W20260812 02:36:28.960105 6046 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0c7c0dfd3fdd4ae0a14f6e96ce847526: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 02:36:28.960240 6046 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 02:36:28.960320 6036 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 02:36:28.961117 6036 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 02:36:28.967810 6036 catalog_manager.cc:1347] Generated new cluster ID: 30748e374794404f8aa42d083281e720
I20260812 02:36:28.967912 6036 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 02:36:29.001320 6036 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 02:36:29.002628 6036 catalog_manager.cc:1504] Loading token signing keys...
I20260812 02:36:29.009795 6036 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0c7c0dfd3fdd4ae0a14f6e96ce847526: Generated new TSK 0
I20260812 02:36:29.010753 6036 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 02:36:29.017206 14748 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbep4M4/build/debug/bin/kudu with pid 5943
2026-08-12T02:36:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1579 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786501898713699-14748-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 02:36:29 dist-test-slave-2w3w krb5kdc[6055](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 02:36:29 dist-test-slave-2w3w krb5kdc[6055](info): set up 2 sockets
Aug 12 02:36:29 dist-test-slave-2w3w krb5kdc[6055](info): commencing operation
krb5kdc: starting...
W20260812 02:36:30.174861 14748 mini_kdc.cc:121] Time spent starting KDC: real 1.135s user 0.005s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786501898713699-14748-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786501898713699-14748-0/krb5kdc/test-user.keytab.
Aug 12 02:36:30 dist-test-slave-2w3w krb5kdc[6055](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502190, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T02:36:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T02:36:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786501898713699-14748-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786501898713699-14748-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 02:36:30.318926 14748 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbep4M4/build/debug/bin/kudu
/tmp/dist-test-taskbep4M4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786501898713699-14748-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786501898713699-14748-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786501898713699-14748-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786501898713699-14748-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34845
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786501898713699-14748-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34845
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786501898713699-14748-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786501898713699-14748-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786501898713699-14748-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:36:30.432530 6073 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:36:30.432878 6073 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:36:30.432971 6073 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:36:30.437063 6073 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:36:30.437165 6073 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:36:30.437206 6073 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:36:30.437266 6073 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:36:30.437307 6073 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 02:36:30.442668 6073 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786501898713699-14748-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786501898713699-14748-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34845
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786501898713699-14748-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34845
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786501898713699-14748-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6073
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786501898713699-14748-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:36:30.443848 6073 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:36:30.444911 6073 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:36:30.453981 6081 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:36:30.453995 6080 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:36:30.454007 6084 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:36:30.454082 6073 server_base.cc:1034] running on GCE node
I20260812 02:36:30.455049 6073 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:36:30.456374 6073 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:36:30.457595 6073 hybrid_clock.cc:648] HybridClock initialized: now 1786502190457566 us; error 49 us; skew 500 ppm
Aug 12 02:36:30 dist-test-slave-2w3w krb5kdc[6055](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502190, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:36:30.460852 6073 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 02:36:30.461944 6073 webserver.cc:460] Webserver started at http://127.0.0.1:34231/ using document root <none> and password file <none>
I20260812 02:36:30.462575 6073 fs_manager.cc:359] Metadata directory not provided
I20260812 02:36:30.462682 6073 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:36:30.462940 6073 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:36:30.465423 6073 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786501898713699-14748-0/minicluster-data/master-0/data/instance:
uuid: "58766c1b985c4f6f8889cfded668f082"
format_stamp: "Formatted at 2026-08-12 02:36:30 on dist-test-slave-2w3w"
server_key: "f9c84b211e09940fedcbf06fbde568a9"
server_key_iv: "c7b39ac2c36b0efe10e7cc916c769014"
server_key_version: "encryptionkey@0"
I20260812 02:36:30.466217 6073 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786501898713699-14748-0/minicluster-data/master-0/wal/instance:
uuid: "58766c1b985c4f6f8889cfded668f082"
format_stamp: "Formatted at 2026-08-12 02:36:30 on dist-test-slave-2w3w"
server_key: "f9c84b211e09940fedcbf06fbde568a9"
server_key_iv: "c7b39ac2c36b0efe10e7cc916c769014"
server_key_version: "encryptionkey@0"
I20260812 02:36:30.470518 6073 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.002s
I20260812 02:36:30.473310 6091 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:36:30.474684 6073 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260812 02:36:30.474895 6073 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786501898713699-14748-0/minicluster-data/master-0/data,/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786501898713699-14748-0/minicluster-data/master-0/wal
uuid: "58766c1b985c4f6f8889cfded668f082"
format_stamp: "Formatted at 2026-08-12 02:36:30 on dist-test-slave-2w3w"
server_key: "f9c84b211e09940fedcbf06fbde568a9"
server_key_iv: "c7b39ac2c36b0efe10e7cc916c769014"
server_key_version: "encryptionkey@0"
I20260812 02:36:30.475106 6073 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786501898713699-14748-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786501898713699-14748-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786501898713699-14748-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:36:30.480185 6073 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:36:30.483839 6073 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:36:30.484082 6073 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:36:30.492537 6073 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34845
I20260812 02:36:30.492549 6146 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34845 every 8 connection(s)
I20260812 02:36:30.493606 6073 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786501898713699-14748-0/minicluster-data/master-0/data/info.pb
I20260812 02:36:30.496704 6147 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:36:30.496941 14748 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbep4M4/build/debug/bin/kudu as pid 6073
I20260812 02:36:30.497092 14748 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786501898713699-14748-0/minicluster-data/master-0/wal/instance
I20260812 02:36:30.497417 14748 external_mini_cluster.cc:1428] Setting key d3e2610b3423be25c7e1da4597cf4283
I20260812 02:36:30.503650 6152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:30.498818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43851 (local address 127.0.0.1:34845)
0812 02:36:30.499350 (+ 532us) server_negotiation.cc:206] Beginning negotiation
0812 02:36:30.499370 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0812 02:36:30.499417 (+ 47us) server_negotiation.cc:407] Connection header received
0812 02:36:30.500515 (+ 1098us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:36:30.500542 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:36:30.500942 (+ 400us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:36:30.501344 (+ 402us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:36:30.502162 (+ 818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:36:30.502167 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:36:30.502181 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:36:30.502195 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:36:30.502261 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:36:30.502288 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:36:30.502438 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0812 02:36:30.502508 (+ 70us) server_negotiation.cc:299] Negotiation successful
0812 02:36:30.502671 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":187,"threads_started":1}
I20260812 02:36:30.504994 6147 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58766c1b985c4f6f8889cfded668f082: Bootstrap starting.
I20260812 02:36:30.506573 14748 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 02:36:30.507611 6147 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 58766c1b985c4f6f8889cfded668f082: Neither blocks nor log segments found. Creating new log.
I20260812 02:36:30.508554 6147 log.cc:824] T 00000000000000000000000000000000 P 58766c1b985c4f6f8889cfded668f082: Log is configured to *not* fsync() on all Append() calls
I20260812 02:36:30.511067 6152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:30.509078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58575 (local address 127.0.0.1:34845)
0812 02:36:30.509229 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0812 02:36:30.509236 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 02:36:30.509414 (+ 178us) server_negotiation.cc:407] Connection header received
0812 02:36:30.509539 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:36:30.509542 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:36:30.509572 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:36:30.509644 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:36:30.510425 (+ 781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:36:30.510428 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:36:30.510429 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:36:30.510440 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:36:30.510480 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:36:30.510491 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:36:30.510562 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0812 02:36:30.510729 (+ 167us) server_negotiation.cc:299] Negotiation successful
0812 02:36:30.510813 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260812 02:36:30.511375 6147 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58766c1b985c4f6f8889cfded668f082: No bootstrap required, opened a new log
I20260812 02:36:30.515484 6147 raft_consensus.cc:348] T 00000000000000000000000000000000 P 58766c1b985c4f6f8889cfded668f082 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58766c1b985c4f6f8889cfded668f082" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34845 } }
I20260812 02:36:30.515803 6147 raft_consensus.cc:374] T 00000000000000000000000000000000 P 58766c1b985c4f6f8889cfded668f082 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:36:30.516016 6147 raft_consensus.cc:729] T 00000000000000000000000000000000 P 58766c1b985c4f6f8889cfded668f082 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58766c1b985c4f6f8889cfded668f082, State: Initialized, Role: FOLLOWER
I20260812 02:36:30.516505 6147 consensus_queue.cc:260] T 00000000000000000000000000000000 P 58766c1b985c4f6f8889cfded668f082 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58766c1b985c4f6f8889cfded668f082" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34845 } }
I20260812 02:36:30.516683 6147 raft_consensus.cc:388] T 00000000000000000000000000000000 P 58766c1b985c4f6f8889cfded668f082 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 02:36:30.516774 6147 raft_consensus.cc:482] T 00000000000000000000000000000000 P 58766c1b985c4f6f8889cfded668f082 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 02:36:30.516891 6147 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 58766c1b985c4f6f8889cfded668f082 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:36:30.517988 6147 raft_consensus.cc:504] T 00000000000000000000000000000000 P 58766c1b985c4f6f8889cfded668f082 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58766c1b985c4f6f8889cfded668f082" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34845 } }
I20260812 02:36:30.518402 6147 leader_election.cc:302] T 00000000000000000000000000000000 P 58766c1b985c4f6f8889cfded668f082 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 58766c1b985c4f6f8889cfded668f082; no voters:
I20260812 02:36:30.518774 6147 leader_election.cc:288] T 00000000000000000000000000000000 P 58766c1b985c4f6f8889cfded668f082 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 02:36:30.518955 6163 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 58766c1b985c4f6f8889cfded668f082 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:36:30.519250 6163 raft_consensus.cc:686] T 00000000000000000000000000000000 P 58766c1b985c4f6f8889cfded668f082 [term 1 LEADER]: Becoming Leader. State: Replica: 58766c1b985c4f6f8889cfded668f082, State: Running, Role: LEADER
I20260812 02:36:30.519642 6163 consensus_queue.cc:237] T 00000000000000000000000000000000 P 58766c1b985c4f6f8889cfded668f082 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58766c1b985c4f6f8889cfded668f082" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34845 } }
I20260812 02:36:30.520227 6147 sys_catalog.cc:564] T 00000000000000000000000000000000 P 58766c1b985c4f6f8889cfded668f082 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:36:30.521533 6165 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58766c1b985c4f6f8889cfded668f082 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 58766c1b985c4f6f8889cfded668f082. Latest consensus state: current_term: 1 leader_uuid: "58766c1b985c4f6f8889cfded668f082" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58766c1b985c4f6f8889cfded668f082" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34845 } } }
I20260812 02:36:30.521536 6164 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58766c1b985c4f6f8889cfded668f082 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "58766c1b985c4f6f8889cfded668f082" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58766c1b985c4f6f8889cfded668f082" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34845 } } }
I20260812 02:36:30.521739 6165 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58766c1b985c4f6f8889cfded668f082 [sys.catalog]: This master's current role is: LEADER
I20260812 02:36:30.521742 6164 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58766c1b985c4f6f8889cfded668f082 [sys.catalog]: This master's current role is: LEADER
I20260812 02:36:30.522187 6172 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 02:36:30.525543 6172 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 02:36:30.531883 6172 catalog_manager.cc:1347] Generated new cluster ID: 7e31176516d14590b3caf1d45c82f830
I20260812 02:36:30.531984 6172 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 02:36:30.565649 6172 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 02:36:30.566048 6172 catalog_manager.cc:1504] Loading token signing keys...
I20260812 02:36:30.575007 6172 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 58766c1b985c4f6f8889cfded668f082: Generated new TSK 0
I20260812 02:36:30.575918 6172 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 02:36:30.583617 14748 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbep4M4/build/debug/bin/kudu with pid 6073
2026-08-12T02:36:30Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1567 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786501898713699-14748-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 02:36:30 dist-test-slave-2w3w krb5kdc[6187](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 02:36:30 dist-test-slave-2w3w krb5kdc[6187](info): set up 2 sockets
Aug 12 02:36:30 dist-test-slave-2w3w krb5kdc[6187](info): commencing operation
krb5kdc: starting...
W20260812 02:36:31.703334 14748 mini_kdc.cc:121] Time spent starting KDC: real 1.096s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786501898713699-14748-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786501898713699-14748-0/krb5kdc/test-user.keytab.
Aug 12 02:36:31 dist-test-slave-2w3w krb5kdc[6187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502191, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T02:36:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T02:36:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786501898713699-14748-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786501898713699-14748-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 02:36:31.840004 14748 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbep4M4/build/debug/bin/kudu
/tmp/dist-test-taskbep4M4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786501898713699-14748-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786501898713699-14748-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786501898713699-14748-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786501898713699-14748-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41959
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786501898713699-14748-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41959
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786501898713699-14748-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786501898713699-14748-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786501898713699-14748-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:36:31.954330 6202 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:36:31.954638 6202 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:36:31.954685 6202 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:36:31.959138 6202 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:36:31.959221 6202 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:36:31.959242 6202 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:36:31.959276 6202 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:36:31.959292 6202 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 02:36:31.965000 6202 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786501898713699-14748-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786501898713699-14748-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41959
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786501898713699-14748-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41959
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786501898713699-14748-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6202
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786501898713699-14748-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:36:31.966212 6202 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:36:31.967446 6202 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:36:31.976632 6210 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:36:31.976613 6208 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:36:31.976611 6207 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:36:31.976812 6202 server_base.cc:1034] running on GCE node
I20260812 02:36:31.977818 6202 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:36:31.979075 6202 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:36:31.980297 6202 hybrid_clock.cc:648] HybridClock initialized: now 1786502191980281 us; error 30 us; skew 500 ppm
Aug 12 02:36:31 dist-test-slave-2w3w krb5kdc[6187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502191, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:36:31.989305 6202 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 02:36:31.990453 6202 webserver.cc:460] Webserver started at http://127.0.0.1:44085/ using document root <none> and password file <none>
I20260812 02:36:31.991184 6202 fs_manager.cc:359] Metadata directory not provided
I20260812 02:36:31.991290 6202 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:36:31.991539 6202 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:36:31.993826 6202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786501898713699-14748-0/minicluster-data/master-0/data/instance:
uuid: "48d2a04c33ca45b5afddf6daf410ba22"
format_stamp: "Formatted at 2026-08-12 02:36:31 on dist-test-slave-2w3w"
server_key: "b8b91b531459f052de79af5b8ae0da6d"
server_key_iv: "cec336cd4916caa13ac50e26428b2e5f"
server_key_version: "encryptionkey@0"
I20260812 02:36:31.994488 6202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786501898713699-14748-0/minicluster-data/master-0/wal/instance:
uuid: "48d2a04c33ca45b5afddf6daf410ba22"
format_stamp: "Formatted at 2026-08-12 02:36:31 on dist-test-slave-2w3w"
server_key: "b8b91b531459f052de79af5b8ae0da6d"
server_key_iv: "cec336cd4916caa13ac50e26428b2e5f"
server_key_version: "encryptionkey@0"
I20260812 02:36:31.998893 6202 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260812 02:36:32.001755 6218 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:36:32.003404 6202 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260812 02:36:32.003577 6202 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786501898713699-14748-0/minicluster-data/master-0/data,/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786501898713699-14748-0/minicluster-data/master-0/wal
uuid: "48d2a04c33ca45b5afddf6daf410ba22"
format_stamp: "Formatted at 2026-08-12 02:36:31 on dist-test-slave-2w3w"
server_key: "b8b91b531459f052de79af5b8ae0da6d"
server_key_iv: "cec336cd4916caa13ac50e26428b2e5f"
server_key_version: "encryptionkey@0"
I20260812 02:36:32.003720 6202 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786501898713699-14748-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786501898713699-14748-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786501898713699-14748-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:36:32.025839 6202 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:36:32.029768 6202 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:36:32.030027 6202 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:36:32.038919 6202 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41959
I20260812 02:36:32.038938 6278 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41959 every 8 connection(s)
I20260812 02:36:32.040036 6202 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786501898713699-14748-0/minicluster-data/master-0/data/info.pb
I20260812 02:36:32.043027 6279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:36:32.048553 14748 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbep4M4/build/debug/bin/kudu as pid 6202
I20260812 02:36:32.048733 14748 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786501898713699-14748-0/minicluster-data/master-0/wal/instance
I20260812 02:36:32.049113 14748 external_mini_cluster.cc:1428] Setting key 929331793e73da78f4538571a0caf047
I20260812 02:36:32.049705 6279 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 48d2a04c33ca45b5afddf6daf410ba22: Bootstrap starting.
I20260812 02:36:32.052460 6279 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 48d2a04c33ca45b5afddf6daf410ba22: Neither blocks nor log segments found. Creating new log.
I20260812 02:36:32.053570 6279 log.cc:824] T 00000000000000000000000000000000 P 48d2a04c33ca45b5afddf6daf410ba22: Log is configured to *not* fsync() on all Append() calls
I20260812 02:36:32.055878 6279 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 48d2a04c33ca45b5afddf6daf410ba22: No bootstrap required, opened a new log
Aug 12 02:36:32 dist-test-slave-2w3w krb5kdc[6187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502191, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260812 02:36:32.060770 6279 raft_consensus.cc:348] T 00000000000000000000000000000000 P 48d2a04c33ca45b5afddf6daf410ba22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48d2a04c33ca45b5afddf6daf410ba22" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41959 } }
I20260812 02:36:32.061050 6279 raft_consensus.cc:374] T 00000000000000000000000000000000 P 48d2a04c33ca45b5afddf6daf410ba22 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:36:32.061143 6279 raft_consensus.cc:729] T 00000000000000000000000000000000 P 48d2a04c33ca45b5afddf6daf410ba22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48d2a04c33ca45b5afddf6daf410ba22, State: Initialized, Role: FOLLOWER
I20260812 02:36:32.061713 6279 consensus_queue.cc:260] T 00000000000000000000000000000000 P 48d2a04c33ca45b5afddf6daf410ba22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48d2a04c33ca45b5afddf6daf410ba22" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41959 } }
I20260812 02:36:32.061903 6279 raft_consensus.cc:388] T 00000000000000000000000000000000 P 48d2a04c33ca45b5afddf6daf410ba22 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 02:36:32.061981 6279 raft_consensus.cc:482] T 00000000000000000000000000000000 P 48d2a04c33ca45b5afddf6daf410ba22 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 02:36:32.062088 6279 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 48d2a04c33ca45b5afddf6daf410ba22 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:36:32.063432 6279 raft_consensus.cc:504] T 00000000000000000000000000000000 P 48d2a04c33ca45b5afddf6daf410ba22 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48d2a04c33ca45b5afddf6daf410ba22" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41959 } }
I20260812 02:36:32.063836 6279 leader_election.cc:302] T 00000000000000000000000000000000 P 48d2a04c33ca45b5afddf6daf410ba22 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 48d2a04c33ca45b5afddf6daf410ba22; no voters:
I20260812 02:36:32.064270 6279 leader_election.cc:288] T 00000000000000000000000000000000 P 48d2a04c33ca45b5afddf6daf410ba22 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 02:36:32.064399 6284 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 48d2a04c33ca45b5afddf6daf410ba22 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:36:32.064707 6284 raft_consensus.cc:686] T 00000000000000000000000000000000 P 48d2a04c33ca45b5afddf6daf410ba22 [term 1 LEADER]: Becoming Leader. State: Replica: 48d2a04c33ca45b5afddf6daf410ba22, State: Running, Role: LEADER
I20260812 02:36:32.065126 6284 consensus_queue.cc:237] T 00000000000000000000000000000000 P 48d2a04c33ca45b5afddf6daf410ba22 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48d2a04c33ca45b5afddf6daf410ba22" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41959 } }
I20260812 02:36:32.065778 6279 sys_catalog.cc:564] T 00000000000000000000000000000000 P 48d2a04c33ca45b5afddf6daf410ba22 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:36:32.066346 6284 sys_catalog.cc:455] T 00000000000000000000000000000000 P 48d2a04c33ca45b5afddf6daf410ba22 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 48d2a04c33ca45b5afddf6daf410ba22. Latest consensus state: current_term: 1 leader_uuid: "48d2a04c33ca45b5afddf6daf410ba22" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48d2a04c33ca45b5afddf6daf410ba22" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41959 } } }
I20260812 02:36:32.066578 6284 sys_catalog.cc:458] T 00000000000000000000000000000000 P 48d2a04c33ca45b5afddf6daf410ba22 [sys.catalog]: This master's current role is: LEADER
I20260812 02:36:32.066818 6287 sys_catalog.cc:455] T 00000000000000000000000000000000 P 48d2a04c33ca45b5afddf6daf410ba22 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "48d2a04c33ca45b5afddf6daf410ba22" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48d2a04c33ca45b5afddf6daf410ba22" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41959 } } }
I20260812 02:36:32.067013 6287 sys_catalog.cc:458] T 00000000000000000000000000000000 P 48d2a04c33ca45b5afddf6daf410ba22 [sys.catalog]: This master's current role is: LEADER
I20260812 02:36:32.067148 6295 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 02:36:32.067354 6282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:32.050698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.84:46521 (local address 127.0.0.1:41959)
0812 02:36:32.051249 (+ 551us) server_negotiation.cc:206] Beginning negotiation
0812 02:36:32.051264 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0812 02:36:32.051302 (+ 38us) server_negotiation.cc:407] Connection header received
0812 02:36:32.052041 (+ 739us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:36:32.052059 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:36:32.052409 (+ 350us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:36:32.052761 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:36:32.053779 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:32.054788 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:32.056121 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:32.056648 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:32.056787 (+ 139us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:36:32.060168 (+ 3381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:36:32.060174 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:36:32.060189 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:36:32.060220 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:36:32.062639 (+ 2419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:36:32.063315 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:36:32.063324 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:36:32.063333 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:36:32.063505 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:36:32.063973 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:36:32.063977 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:36:32.063979 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:36:32.064624 (+ 645us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:36:32.065634 (+ 1010us) server_negotiation.cc:1035] Waiting for connection context
0812 02:36:32.065840 (+ 206us) server_negotiation.cc:299] Negotiation successful
0812 02:36:32.066097 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":393,"thread_start_us":188,"threads_started":1}
I20260812 02:36:32.070950 6295 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 02:36:32.071527 14748 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 02:36:32.079711 6295 catalog_manager.cc:1347] Generated new cluster ID: b733aabc46c74a068e5ec84f2df203d1
I20260812 02:36:32.079807 6295 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 02:36:32.090173 6282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:32.075407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.84:58423 (local address 127.0.0.1:41959)
0812 02:36:32.075547 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0812 02:36:32.075551 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:36:32.075817 (+ 266us) server_negotiation.cc:407] Connection header received
0812 02:36:32.076056 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:36:32.076060 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:36:32.076119 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:36:32.076181 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:36:32.077744 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:32.078637 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:32.083583 (+ 4946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:32.084053 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:32.084317 (+ 264us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:36:32.086238 (+ 1921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:36:32.086242 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:36:32.086246 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:36:32.086278 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:36:32.088149 (+ 1871us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:36:32.088760 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:36:32.088764 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:36:32.088766 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:36:32.088860 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:36:32.089212 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:36:32.089216 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:36:32.089217 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:36:32.089443 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:36:32.089588 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0812 02:36:32.089877 (+ 289us) server_negotiation.cc:299] Negotiation successful
0812 02:36:32.089994 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260812 02:36:32.098531 6295 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 02:36:32.100034 6295 catalog_manager.cc:1504] Loading token signing keys...
I20260812 02:36:32.108471 6295 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 48d2a04c33ca45b5afddf6daf410ba22: Generated new TSK 0
I20260812 02:36:32.109366 6295 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 02:36:32.117316 14748 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbep4M4/build/debug/bin/kudu with pid 6202
W20260812 02:36:32.126798 6184 connection.cc:535] client connection to 127.0.0.1:41959 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41959): Connection reset by peer (error 104)
2026-08-12T02:36:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1534 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786501898713699-14748-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 02:36:32 dist-test-slave-2w3w krb5kdc[6318](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 02:36:32 dist-test-slave-2w3w krb5kdc[6318](info): set up 2 sockets
Aug 12 02:36:32 dist-test-slave-2w3w krb5kdc[6318](info): commencing operation
krb5kdc: starting...
W20260812 02:36:33.242003 14748 mini_kdc.cc:121] Time spent starting KDC: real 1.100s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786501898713699-14748-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786501898713699-14748-0/krb5kdc/test-user.keytab.
Aug 12 02:36:33 dist-test-slave-2w3w krb5kdc[6318](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502193, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T02:36:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T02:36:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786501898713699-14748-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786501898713699-14748-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 02:36:33.381707 14748 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbep4M4/build/debug/bin/kudu
/tmp/dist-test-taskbep4M4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786501898713699-14748-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786501898713699-14748-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786501898713699-14748-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786501898713699-14748-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32783
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786501898713699-14748-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:32783
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786501898713699-14748-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786501898713699-14748-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786501898713699-14748-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:36:33.499459 6333 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:36:33.499814 6333 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:36:33.499908 6333 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:36:33.504171 6333 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:36:33.504299 6333 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:36:33.504334 6333 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:36:33.504364 6333 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:36:33.504402 6333 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 02:36:33.510280 6333 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786501898713699-14748-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786501898713699-14748-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:32783
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786501898713699-14748-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32783
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786501898713699-14748-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6333
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786501898713699-14748-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:36:33.511652 6333 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:36:33.512818 6333 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:36:33.521943 6338 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:36:33.522020 6339 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:36:33.521943 6342 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:36:33.522401 6333 server_base.cc:1034] running on GCE node
I20260812 02:36:33.523043 6333 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:36:33.524283 6333 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:36:33.525556 6333 hybrid_clock.cc:648] HybridClock initialized: now 1786502193525543 us; error 46 us; skew 500 ppm
Aug 12 02:36:33 dist-test-slave-2w3w krb5kdc[6318](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502193, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:36:33.528760 6333 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 02:36:33.529834 6333 webserver.cc:460] Webserver started at http://127.0.0.1:40171/ using document root <none> and password file <none>
I20260812 02:36:33.530432 6333 fs_manager.cc:359] Metadata directory not provided
I20260812 02:36:33.530529 6333 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:36:33.530774 6333 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:36:33.533005 6333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786501898713699-14748-0/minicluster-data/master-0/data/instance:
uuid: "bfb269fa8ce54150a01d0f9eb3c562c5"
format_stamp: "Formatted at 2026-08-12 02:36:33 on dist-test-slave-2w3w"
server_key: "3d6c1719202bd4b74335e9411453f5d2"
server_key_iv: "1ff6b5a63ca483239fbad414eb1e98b3"
server_key_version: "encryptionkey@0"
I20260812 02:36:33.533617 6333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786501898713699-14748-0/minicluster-data/master-0/wal/instance:
uuid: "bfb269fa8ce54150a01d0f9eb3c562c5"
format_stamp: "Formatted at 2026-08-12 02:36:33 on dist-test-slave-2w3w"
server_key: "3d6c1719202bd4b74335e9411453f5d2"
server_key_iv: "1ff6b5a63ca483239fbad414eb1e98b3"
server_key_version: "encryptionkey@0"
I20260812 02:36:33.537772 6333 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260812 02:36:33.540791 6353 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:36:33.542405 6333 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260812 02:36:33.542608 6333 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786501898713699-14748-0/minicluster-data/master-0/data,/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786501898713699-14748-0/minicluster-data/master-0/wal
uuid: "bfb269fa8ce54150a01d0f9eb3c562c5"
format_stamp: "Formatted at 2026-08-12 02:36:33 on dist-test-slave-2w3w"
server_key: "3d6c1719202bd4b74335e9411453f5d2"
server_key_iv: "1ff6b5a63ca483239fbad414eb1e98b3"
server_key_version: "encryptionkey@0"
I20260812 02:36:33.542761 6333 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786501898713699-14748-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786501898713699-14748-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786501898713699-14748-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:36:33.557334 6333 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:36:33.560680 6333 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:36:33.560907 6333 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:36:33.568521 6333 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:32783
I20260812 02:36:33.568539 6413 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:32783 every 8 connection(s)
I20260812 02:36:33.569564 6333 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786501898713699-14748-0/minicluster-data/master-0/data/info.pb
I20260812 02:36:33.569944 14748 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbep4M4/build/debug/bin/kudu as pid 6333
I20260812 02:36:33.570122 14748 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786501898713699-14748-0/minicluster-data/master-0/wal/instance
I20260812 02:36:33.570436 14748 external_mini_cluster.cc:1428] Setting key 17463d330a01fe9d691fc36b3e79dff8
I20260812 02:36:33.572957 6414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:36:33.579566 6414 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bfb269fa8ce54150a01d0f9eb3c562c5: Bootstrap starting.
I20260812 02:36:33.580049 6417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:33.572029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.84:56511 (local address 127.0.0.1:32783)
0812 02:36:33.572488 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0812 02:36:33.572503 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0812 02:36:33.572538 (+ 35us) server_negotiation.cc:407] Connection header received
0812 02:36:33.573284 (+ 746us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:36:33.573304 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:36:33.573651 (+ 347us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:36:33.573943 (+ 292us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:36:33.574287 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:33.575408 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:33.576648 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:33.577116 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:33.577240 (+ 124us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:36:33.578210 (+ 970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:36:33.578214 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:36:33.578227 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:36:33.578623 (+ 396us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:36:33.578677 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:36:33.578700 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:36:33.578806 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0812 02:36:33.579116 (+ 310us) server_negotiation.cc:299] Negotiation successful
0812 02:36:33.579267 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":149,"threads_started":1}
I20260812 02:36:33.582087 6414 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bfb269fa8ce54150a01d0f9eb3c562c5: Neither blocks nor log segments found. Creating new log.
I20260812 02:36:33.583003 6414 log.cc:824] T 00000000000000000000000000000000 P bfb269fa8ce54150a01d0f9eb3c562c5: Log is configured to *not* fsync() on all Append() calls
I20260812 02:36:33.583132 14748 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 02:36:33.585351 6414 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bfb269fa8ce54150a01d0f9eb3c562c5: No bootstrap required, opened a new log
I20260812 02:36:33.589802 6417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:33.585322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.84:55439 (local address 127.0.0.1:32783)
0812 02:36:33.585458 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0812 02:36:33.585462 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:36:33.585691 (+ 229us) server_negotiation.cc:407] Connection header received
0812 02:36:33.585766 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:36:33.585769 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:36:33.585817 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:36:33.585932 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:36:33.586186 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:33.586984 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:33.587854 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:33.588165 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:33.588276 (+ 111us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:36:33.589264 (+ 988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:36:33.589268 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:36:33.589270 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:36:33.589287 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:36:33.589326 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:36:33.589340 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:36:33.589434 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0812 02:36:33.589625 (+ 191us) server_negotiation.cc:299] Negotiation successful
0812 02:36:33.589666 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260812 02:36:33.589953 6414 raft_consensus.cc:348] T 00000000000000000000000000000000 P bfb269fa8ce54150a01d0f9eb3c562c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bfb269fa8ce54150a01d0f9eb3c562c5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32783 } }
I20260812 02:36:33.590268 6414 raft_consensus.cc:374] T 00000000000000000000000000000000 P bfb269fa8ce54150a01d0f9eb3c562c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:36:33.590343 6414 raft_consensus.cc:729] T 00000000000000000000000000000000 P bfb269fa8ce54150a01d0f9eb3c562c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bfb269fa8ce54150a01d0f9eb3c562c5, State: Initialized, Role: FOLLOWER
I20260812 02:36:33.590896 6414 consensus_queue.cc:260] T 00000000000000000000000000000000 P bfb269fa8ce54150a01d0f9eb3c562c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bfb269fa8ce54150a01d0f9eb3c562c5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32783 } }
I20260812 02:36:33.591131 6414 raft_consensus.cc:388] T 00000000000000000000000000000000 P bfb269fa8ce54150a01d0f9eb3c562c5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 02:36:33.591207 6414 raft_consensus.cc:482] T 00000000000000000000000000000000 P bfb269fa8ce54150a01d0f9eb3c562c5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 02:36:33.591311 6414 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bfb269fa8ce54150a01d0f9eb3c562c5 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:36:33.592344 6414 raft_consensus.cc:504] T 00000000000000000000000000000000 P bfb269fa8ce54150a01d0f9eb3c562c5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bfb269fa8ce54150a01d0f9eb3c562c5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32783 } }
I20260812 02:36:33.592708 6414 leader_election.cc:302] T 00000000000000000000000000000000 P bfb269fa8ce54150a01d0f9eb3c562c5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bfb269fa8ce54150a01d0f9eb3c562c5; no voters:
I20260812 02:36:33.593045 6414 leader_election.cc:288] T 00000000000000000000000000000000 P bfb269fa8ce54150a01d0f9eb3c562c5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 02:36:33.593192 6425 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bfb269fa8ce54150a01d0f9eb3c562c5 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:36:33.593442 6425 raft_consensus.cc:686] T 00000000000000000000000000000000 P bfb269fa8ce54150a01d0f9eb3c562c5 [term 1 LEADER]: Becoming Leader. State: Replica: bfb269fa8ce54150a01d0f9eb3c562c5, State: Running, Role: LEADER
I20260812 02:36:33.593788 6425 consensus_queue.cc:237] T 00000000000000000000000000000000 P bfb269fa8ce54150a01d0f9eb3c562c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bfb269fa8ce54150a01d0f9eb3c562c5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32783 } }
I20260812 02:36:33.594349 6414 sys_catalog.cc:564] T 00000000000000000000000000000000 P bfb269fa8ce54150a01d0f9eb3c562c5 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:36:33.595639 6426 sys_catalog.cc:455] T 00000000000000000000000000000000 P bfb269fa8ce54150a01d0f9eb3c562c5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bfb269fa8ce54150a01d0f9eb3c562c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bfb269fa8ce54150a01d0f9eb3c562c5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32783 } } }
I20260812 02:36:33.595767 6426 sys_catalog.cc:458] T 00000000000000000000000000000000 P bfb269fa8ce54150a01d0f9eb3c562c5 [sys.catalog]: This master's current role is: LEADER
I20260812 02:36:33.596132 6427 sys_catalog.cc:455] T 00000000000000000000000000000000 P bfb269fa8ce54150a01d0f9eb3c562c5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bfb269fa8ce54150a01d0f9eb3c562c5. Latest consensus state: current_term: 1 leader_uuid: "bfb269fa8ce54150a01d0f9eb3c562c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bfb269fa8ce54150a01d0f9eb3c562c5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32783 } } }
I20260812 02:36:33.596298 6434 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 02:36:33.596304 6427 sys_catalog.cc:458] T 00000000000000000000000000000000 P bfb269fa8ce54150a01d0f9eb3c562c5 [sys.catalog]: This master's current role is: LEADER
I20260812 02:36:33.599172 6434 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 02:36:33.604519 6434 catalog_manager.cc:1347] Generated new cluster ID: 77894284ab0a4250b0853cd89c0cfc45
I20260812 02:36:33.604604 6434 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 02:36:33.618862 6434 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 02:36:33.619905 6434 catalog_manager.cc:1504] Loading token signing keys...
I20260812 02:36:33.628404 6434 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bfb269fa8ce54150a01d0f9eb3c562c5: Generated new TSK 0
I20260812 02:36:33.629182 6434 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 02:36:33.636152 14748 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbep4M4/build/debug/bin/kudu with pid 6333
W20260812 02:36:33.643877 6316 connection.cc:535] client connection to 127.0.0.1:32783 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:32783): Connection reset by peer (error 104)
2026-08-12T02:36:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1516 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786501898713699-14748-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 02:36:33 dist-test-slave-2w3w krb5kdc[6449](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 02:36:33 dist-test-slave-2w3w krb5kdc[6449](info): set up 2 sockets
Aug 12 02:36:33 dist-test-slave-2w3w krb5kdc[6449](info): commencing operation
krb5kdc: starting...
W20260812 02:36:34.753144 14748 mini_kdc.cc:121] Time spent starting KDC: real 1.095s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786501898713699-14748-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786501898713699-14748-0/krb5kdc/test-user.keytab.
Aug 12 02:36:34 dist-test-slave-2w3w krb5kdc[6449](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502194, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T02:36:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T02:36:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786501898713699-14748-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786501898713699-14748-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 02:36:34.903847 14748 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbep4M4/build/debug/bin/kudu
/tmp/dist-test-taskbep4M4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786501898713699-14748-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786501898713699-14748-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786501898713699-14748-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786501898713699-14748-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37357
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786501898713699-14748-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37357
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786501898713699-14748-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786501898713699-14748-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786501898713699-14748-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:36:35.024080 6466 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:36:35.024438 6466 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:36:35.024545 6466 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:36:35.028854 6466 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:36:35.028970 6466 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:36:35.029009 6466 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:36:35.029071 6466 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:36:35.029126 6466 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 02:36:35.035167 6466 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786501898713699-14748-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786501898713699-14748-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37357
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786501898713699-14748-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37357
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786501898713699-14748-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6466
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786501898713699-14748-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:36:35.036444 6466 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:36:35.037663 6466 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:36:35.047369 6471 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:36:35.047488 6472 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:36:35.047376 6474 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:36:35.049256 6466 server_base.cc:1034] running on GCE node
I20260812 02:36:35.050019 6466 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:36:35.051796 6466 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:36:35.053058 6466 hybrid_clock.cc:648] HybridClock initialized: now 1786502195053039 us; error 48 us; skew 500 ppm
Aug 12 02:36:35 dist-test-slave-2w3w krb5kdc[6449](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502195, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:36:35.056349 6466 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 02:36:35.057332 6466 webserver.cc:460] Webserver started at http://127.0.0.1:38113/ using document root <none> and password file <none>
I20260812 02:36:35.058015 6466 fs_manager.cc:359] Metadata directory not provided
I20260812 02:36:35.058115 6466 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:36:35.058353 6466 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:36:35.060582 6466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786501898713699-14748-0/minicluster-data/master-0/data/instance:
uuid: "ad9c66afc0e94823ac31077f291c6285"
format_stamp: "Formatted at 2026-08-12 02:36:35 on dist-test-slave-2w3w"
server_key: "7c52961be331ef82482d6703aca71b38"
server_key_iv: "f1d86fb47b025610bb17dee12ddcfd8d"
server_key_version: "encryptionkey@0"
I20260812 02:36:35.061211 6466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786501898713699-14748-0/minicluster-data/master-0/wal/instance:
uuid: "ad9c66afc0e94823ac31077f291c6285"
format_stamp: "Formatted at 2026-08-12 02:36:35 on dist-test-slave-2w3w"
server_key: "7c52961be331ef82482d6703aca71b38"
server_key_iv: "f1d86fb47b025610bb17dee12ddcfd8d"
server_key_version: "encryptionkey@0"
I20260812 02:36:35.065317 6466 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260812 02:36:35.067953 6482 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:36:35.069288 6466 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260812 02:36:35.069446 6466 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786501898713699-14748-0/minicluster-data/master-0/data,/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786501898713699-14748-0/minicluster-data/master-0/wal
uuid: "ad9c66afc0e94823ac31077f291c6285"
format_stamp: "Formatted at 2026-08-12 02:36:35 on dist-test-slave-2w3w"
server_key: "7c52961be331ef82482d6703aca71b38"
server_key_iv: "f1d86fb47b025610bb17dee12ddcfd8d"
server_key_version: "encryptionkey@0"
I20260812 02:36:35.069571 6466 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786501898713699-14748-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786501898713699-14748-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786501898713699-14748-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:36:35.074741 6466 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:36:35.078390 6466 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:36:35.078620 6466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:36:35.086763 6466 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37357
I20260812 02:36:35.086793 6547 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37357 every 8 connection(s)
I20260812 02:36:35.088148 6466 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786501898713699-14748-0/minicluster-data/master-0/data/info.pb
I20260812 02:36:35.091638 6548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:36:35.092299 14748 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbep4M4/build/debug/bin/kudu as pid 6466
I20260812 02:36:35.092468 14748 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786501898713699-14748-0/minicluster-data/master-0/wal/instance
I20260812 02:36:35.092761 14748 external_mini_cluster.cc:1428] Setting key 5678bc31c91bc5a862074d29868d3112
I20260812 02:36:35.098709 6548 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad9c66afc0e94823ac31077f291c6285: Bootstrap starting.
I20260812 02:36:35.099027 6551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:35.094202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.84:35711 (local address 127.0.0.1:37357)
0812 02:36:35.094696 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0812 02:36:35.094718 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0812 02:36:35.095355 (+ 637us) server_negotiation.cc:407] Connection header received
0812 02:36:35.096113 (+ 758us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:36:35.096132 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:36:35.096438 (+ 306us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:36:35.096722 (+ 284us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:36:35.097656 (+ 934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:36:35.097661 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:36:35.097673 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:36:35.097683 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:36:35.097736 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:36:35.097755 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:36:35.097877 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0812 02:36:35.097995 (+ 118us) server_negotiation.cc:299] Negotiation successful
0812 02:36:35.098169 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":142,"threads_started":1}
I20260812 02:36:35.101511 6548 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ad9c66afc0e94823ac31077f291c6285: Neither blocks nor log segments found. Creating new log.
I20260812 02:36:35.102067 14748 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 02:36:35.102439 6548 log.cc:824] T 00000000000000000000000000000000 P ad9c66afc0e94823ac31077f291c6285: Log is configured to *not* fsync() on all Append() calls
I20260812 02:36:35.104786 6548 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad9c66afc0e94823ac31077f291c6285: No bootstrap required, opened a new log
I20260812 02:36:35.106076 6551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:35.104323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.84:42949 (local address 127.0.0.1:37357)
0812 02:36:35.104466 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0812 02:36:35.104471 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:36:35.104644 (+ 173us) server_negotiation.cc:407] Connection header received
0812 02:36:35.104745 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:36:35.104748 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:36:35.104784 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:36:35.104870 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:36:35.105603 (+ 733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:36:35.105607 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:36:35.105610 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:36:35.105624 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:36:35.105656 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:36:35.105669 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:36:35.105766 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0812 02:36:35.105896 (+ 130us) server_negotiation.cc:299] Negotiation successful
0812 02:36:35.105947 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260812 02:36:35.108944 6548 raft_consensus.cc:348] T 00000000000000000000000000000000 P ad9c66afc0e94823ac31077f291c6285 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad9c66afc0e94823ac31077f291c6285" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37357 } }
I20260812 02:36:35.109186 6548 raft_consensus.cc:374] T 00000000000000000000000000000000 P ad9c66afc0e94823ac31077f291c6285 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:36:35.109221 6548 raft_consensus.cc:729] T 00000000000000000000000000000000 P ad9c66afc0e94823ac31077f291c6285 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad9c66afc0e94823ac31077f291c6285, State: Initialized, Role: FOLLOWER
I20260812 02:36:35.109642 6548 consensus_queue.cc:260] T 00000000000000000000000000000000 P ad9c66afc0e94823ac31077f291c6285 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad9c66afc0e94823ac31077f291c6285" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37357 } }
I20260812 02:36:35.109767 6548 raft_consensus.cc:388] T 00000000000000000000000000000000 P ad9c66afc0e94823ac31077f291c6285 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 02:36:35.109809 6548 raft_consensus.cc:482] T 00000000000000000000000000000000 P ad9c66afc0e94823ac31077f291c6285 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 02:36:35.109884 6548 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ad9c66afc0e94823ac31077f291c6285 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:36:35.110888 6548 raft_consensus.cc:504] T 00000000000000000000000000000000 P ad9c66afc0e94823ac31077f291c6285 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad9c66afc0e94823ac31077f291c6285" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37357 } }
I20260812 02:36:35.111265 6548 leader_election.cc:302] T 00000000000000000000000000000000 P ad9c66afc0e94823ac31077f291c6285 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ad9c66afc0e94823ac31077f291c6285; no voters:
I20260812 02:36:35.111614 6548 leader_election.cc:288] T 00000000000000000000000000000000 P ad9c66afc0e94823ac31077f291c6285 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 02:36:35.111788 6558 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ad9c66afc0e94823ac31077f291c6285 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:36:35.112222 6558 raft_consensus.cc:686] T 00000000000000000000000000000000 P ad9c66afc0e94823ac31077f291c6285 [term 1 LEADER]: Becoming Leader. State: Replica: ad9c66afc0e94823ac31077f291c6285, State: Running, Role: LEADER
I20260812 02:36:35.112569 6558 consensus_queue.cc:237] T 00000000000000000000000000000000 P ad9c66afc0e94823ac31077f291c6285 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad9c66afc0e94823ac31077f291c6285" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37357 } }
I20260812 02:36:35.113021 6548 sys_catalog.cc:564] T 00000000000000000000000000000000 P ad9c66afc0e94823ac31077f291c6285 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:36:35.114348 6560 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad9c66afc0e94823ac31077f291c6285 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ad9c66afc0e94823ac31077f291c6285. Latest consensus state: current_term: 1 leader_uuid: "ad9c66afc0e94823ac31077f291c6285" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad9c66afc0e94823ac31077f291c6285" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37357 } } }
I20260812 02:36:35.114509 6560 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad9c66afc0e94823ac31077f291c6285 [sys.catalog]: This master's current role is: LEADER
I20260812 02:36:35.114842 6559 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad9c66afc0e94823ac31077f291c6285 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ad9c66afc0e94823ac31077f291c6285" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad9c66afc0e94823ac31077f291c6285" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37357 } } }
I20260812 02:36:35.114952 6559 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad9c66afc0e94823ac31077f291c6285 [sys.catalog]: This master's current role is: LEADER
I20260812 02:36:35.115293 6568 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 02:36:35.118252 6568 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 02:36:35.124917 6568 catalog_manager.cc:1347] Generated new cluster ID: 67fea86168f445fc91c8a956ab02fec0
I20260812 02:36:35.125020 6568 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 02:36:35.133891 6568 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 02:36:35.134369 6568 catalog_manager.cc:1504] Loading token signing keys...
I20260812 02:36:35.141487 6568 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ad9c66afc0e94823ac31077f291c6285: Generated new TSK 0
I20260812 02:36:35.142483 6568 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 02:36:35.146091 14748 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbep4M4/build/debug/bin/kudu with pid 6466
2026-08-12T02:36:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1512 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786501898713699-14748-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 02:36:35 dist-test-slave-2w3w krb5kdc[6579](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 02:36:35 dist-test-slave-2w3w krb5kdc[6579](info): set up 2 sockets
Aug 12 02:36:35 dist-test-slave-2w3w krb5kdc[6579](info): commencing operation
krb5kdc: starting...
W20260812 02:36:36.278020 14748 mini_kdc.cc:121] Time spent starting KDC: real 1.107s user 0.000s sys 0.009s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786501898713699-14748-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786501898713699-14748-0/krb5kdc/test-user.keytab.
Aug 12 02:36:36 dist-test-slave-2w3w krb5kdc[6579](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502196, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T02:36:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T02:36:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786501898713699-14748-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786501898713699-14748-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 02:36:36.418706 14748 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbep4M4/build/debug/bin/kudu
/tmp/dist-test-taskbep4M4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786501898713699-14748-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786501898713699-14748-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786501898713699-14748-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786501898713699-14748-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35019
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786501898713699-14748-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37333
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35019
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786501898713699-14748-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786501898713699-14748-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786501898713699-14748-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:36:36.534788 6594 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:36:36.535220 6594 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:36:36.535326 6594 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:36:36.540171 6594 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:36:36.540264 6594 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:36:36.540285 6594 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:36:36.540303 6594 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:36:36.540321 6594 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 02:36:36.546021 6594 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37333
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786501898713699-14748-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786501898713699-14748-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35019
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786501898713699-14748-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35019
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786501898713699-14748-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6594
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786501898713699-14748-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:36:36.547191 6594 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:36:36.548184 6594 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:36:36.557106 6603 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:36:36.557092 6605 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:36:36.557092 6602 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:36:36.557680 6594 server_base.cc:1034] running on GCE node
I20260812 02:36:36.558315 6594 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:36:36.559574 6594 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:36:36.560792 6594 hybrid_clock.cc:648] HybridClock initialized: now 1786502196560767 us; error 39 us; skew 500 ppm
Aug 12 02:36:36 dist-test-slave-2w3w krb5kdc[6579](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502196, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:36:36.563838 6594 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 02:36:36.564713 6594 webserver.cc:460] Webserver started at http://127.0.0.1:46827/ using document root <none> and password file <none>
I20260812 02:36:36.565237 6594 fs_manager.cc:359] Metadata directory not provided
I20260812 02:36:36.565296 6594 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:36:36.565475 6594 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:36:36.567647 6594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786501898713699-14748-0/minicluster-data/master-0/data/instance:
uuid: "ec06dff49c9b4f49840e143329cd39c0"
format_stamp: "Formatted at 2026-08-12 02:36:36 on dist-test-slave-2w3w"
server_key: "5390da5665afa31086b13635d90e4526"
server_key_iv: "8fb5f8cc659c70e9f6bdf671c52e2558"
server_key_version: "encryptionkey@0"
I20260812 02:36:36.568169 6594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786501898713699-14748-0/minicluster-data/master-0/wal/instance:
uuid: "ec06dff49c9b4f49840e143329cd39c0"
format_stamp: "Formatted at 2026-08-12 02:36:36 on dist-test-slave-2w3w"
server_key: "5390da5665afa31086b13635d90e4526"
server_key_iv: "8fb5f8cc659c70e9f6bdf671c52e2558"
server_key_version: "encryptionkey@0"
I20260812 02:36:36.571950 6594 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260812 02:36:36.574317 6614 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:36:36.575811 6594 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260812 02:36:36.575932 6594 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786501898713699-14748-0/minicluster-data/master-0/data,/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786501898713699-14748-0/minicluster-data/master-0/wal
uuid: "ec06dff49c9b4f49840e143329cd39c0"
format_stamp: "Formatted at 2026-08-12 02:36:36 on dist-test-slave-2w3w"
server_key: "5390da5665afa31086b13635d90e4526"
server_key_iv: "8fb5f8cc659c70e9f6bdf671c52e2558"
server_key_version: "encryptionkey@0"
I20260812 02:36:36.576033 6594 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786501898713699-14748-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786501898713699-14748-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786501898713699-14748-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:36:36.616756 6594 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:36:36.620798 6594 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:36:36.621062 6594 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:36:36.629434 6594 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35019
I20260812 02:36:36.629483 6671 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35019 every 8 connection(s)
I20260812 02:36:36.630517 6594 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786501898713699-14748-0/minicluster-data/master-0/data/info.pb
I20260812 02:36:36.633694 6672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:36:36.637364 14748 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbep4M4/build/debug/bin/kudu as pid 6594
I20260812 02:36:36.637554 14748 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786501898713699-14748-0/minicluster-data/master-0/wal/instance
I20260812 02:36:36.637852 14748 external_mini_cluster.cc:1428] Setting key 79baf07c4f85893aac9b1c1ff3246f0c
I20260812 02:36:36.640914 6672 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec06dff49c9b4f49840e143329cd39c0: Bootstrap starting.
I20260812 02:36:36.644060 6672 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ec06dff49c9b4f49840e143329cd39c0: Neither blocks nor log segments found. Creating new log.
I20260812 02:36:36.645066 6672 log.cc:824] T 00000000000000000000000000000000 P ec06dff49c9b4f49840e143329cd39c0: Log is configured to *not* fsync() on all Append() calls
I20260812 02:36:36.647415 6672 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec06dff49c9b4f49840e143329cd39c0: No bootstrap required, opened a new log
Aug 12 02:36:36 dist-test-slave-2w3w krb5kdc[6579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502196, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260812 02:36:36.652060 6672 raft_consensus.cc:348] T 00000000000000000000000000000000 P ec06dff49c9b4f49840e143329cd39c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec06dff49c9b4f49840e143329cd39c0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35019 } }
I20260812 02:36:36.652372 6672 raft_consensus.cc:374] T 00000000000000000000000000000000 P ec06dff49c9b4f49840e143329cd39c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:36:36.652454 6672 raft_consensus.cc:729] T 00000000000000000000000000000000 P ec06dff49c9b4f49840e143329cd39c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec06dff49c9b4f49840e143329cd39c0, State: Initialized, Role: FOLLOWER
I20260812 02:36:36.653003 6672 consensus_queue.cc:260] T 00000000000000000000000000000000 P ec06dff49c9b4f49840e143329cd39c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec06dff49c9b4f49840e143329cd39c0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35019 } }
I20260812 02:36:36.653146 6672 raft_consensus.cc:388] T 00000000000000000000000000000000 P ec06dff49c9b4f49840e143329cd39c0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 02:36:36.653237 6672 raft_consensus.cc:482] T 00000000000000000000000000000000 P ec06dff49c9b4f49840e143329cd39c0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 02:36:36.653371 6672 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ec06dff49c9b4f49840e143329cd39c0 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:36:36.654551 6672 raft_consensus.cc:504] T 00000000000000000000000000000000 P ec06dff49c9b4f49840e143329cd39c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec06dff49c9b4f49840e143329cd39c0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35019 } }
I20260812 02:36:36.654997 6672 leader_election.cc:302] T 00000000000000000000000000000000 P ec06dff49c9b4f49840e143329cd39c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ec06dff49c9b4f49840e143329cd39c0; no voters:
I20260812 02:36:36.655272 6677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:36.639683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.84:39783 (local address 127.0.0.1:35019)
0812 02:36:36.640180 (+ 497us) server_negotiation.cc:206] Beginning negotiation
0812 02:36:36.640203 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0812 02:36:36.640249 (+ 46us) server_negotiation.cc:407] Connection header received
0812 02:36:36.641225 (+ 976us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:36:36.641250 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:36:36.641738 (+ 488us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:36:36.642136 (+ 398us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:36:36.642927 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:36.644668 (+ 1741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:36.645714 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:36.646245 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:36.646448 (+ 203us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:36:36.649551 (+ 3103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:36:36.649556 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:36:36.649570 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:36:36.649601 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:36:36.651659 (+ 2058us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:36:36.652258 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:36:36.652264 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:36:36.652269 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:36:36.652444 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:36:36.652851 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:36:36.652854 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:36:36.652856 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:36:36.653432 (+ 576us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:36:36.653669 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0812 02:36:36.654029 (+ 360us) server_negotiation.cc:299] Negotiation successful
0812 02:36:36.654275 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":11,"server-negotiator.queue_time_us":332,"thread_start_us":144,"threads_started":1}
I20260812 02:36:36.655417 6672 leader_election.cc:288] T 00000000000000000000000000000000 P ec06dff49c9b4f49840e143329cd39c0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 02:36:36.655718 6679 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ec06dff49c9b4f49840e143329cd39c0 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:36:36.655972 6679 raft_consensus.cc:686] T 00000000000000000000000000000000 P ec06dff49c9b4f49840e143329cd39c0 [term 1 LEADER]: Becoming Leader. State: Replica: ec06dff49c9b4f49840e143329cd39c0, State: Running, Role: LEADER
I20260812 02:36:36.656327 6679 consensus_queue.cc:237] T 00000000000000000000000000000000 P ec06dff49c9b4f49840e143329cd39c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec06dff49c9b4f49840e143329cd39c0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35019 } }
I20260812 02:36:36.656944 6672 sys_catalog.cc:564] T 00000000000000000000000000000000 P ec06dff49c9b4f49840e143329cd39c0 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:36:36.657685 6679 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec06dff49c9b4f49840e143329cd39c0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ec06dff49c9b4f49840e143329cd39c0. Latest consensus state: current_term: 1 leader_uuid: "ec06dff49c9b4f49840e143329cd39c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec06dff49c9b4f49840e143329cd39c0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35019 } } }
I20260812 02:36:36.657837 6679 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec06dff49c9b4f49840e143329cd39c0 [sys.catalog]: This master's current role is: LEADER
I20260812 02:36:36.658577 14748 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 02:36:36.658834 6681 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec06dff49c9b4f49840e143329cd39c0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ec06dff49c9b4f49840e143329cd39c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec06dff49c9b4f49840e143329cd39c0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35019 } } }
I20260812 02:36:36.658929 6681 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec06dff49c9b4f49840e143329cd39c0 [sys.catalog]: This master's current role is: LEADER
W20260812 02:36:36.662819 6696 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ec06dff49c9b4f49840e143329cd39c0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 02:36:36.662948 6696 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 02:36:36.663066 6687 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 02:36:36.663861 6687 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 02:36:36.670469 6687 catalog_manager.cc:1347] Generated new cluster ID: dbb57eb7bacc4ba9b15fe6ea231c7c89
I20260812 02:36:36.670562 6687 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 02:36:36.677186 6677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:36.661285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.84:56655 (local address 127.0.0.1:35019)
0812 02:36:36.661432 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0812 02:36:36.661437 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:36:36.667423 (+ 5986us) server_negotiation.cc:407] Connection header received
0812 02:36:36.667604 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:36:36.667608 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:36:36.667663 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:36:36.667759 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:36:36.668713 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:36.669542 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:36.670989 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:36.671340 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:36.671456 (+ 116us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:36:36.673345 (+ 1889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:36:36.673349 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:36:36.673352 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:36:36.673380 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:36:36.675145 (+ 1765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:36:36.675784 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:36:36.675787 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:36:36.675789 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:36:36.675878 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:36:36.676232 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:36:36.676235 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:36:36.676237 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:36:36.676454 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:36:36.676615 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0812 02:36:36.676908 (+ 293us) server_negotiation.cc:299] Negotiation successful
0812 02:36:36.677016 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260812 02:36:36.691112 6687 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 02:36:36.692466 6687 catalog_manager.cc:1504] Loading token signing keys...
I20260812 02:36:36.702316 6687 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ec06dff49c9b4f49840e143329cd39c0: Generated new TSK 0
I20260812 02:36:36.703291 6687 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 02:36:36.712236 14748 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbep4M4/build/debug/bin/kudu with pid 6594
W20260812 02:36:36.722524 6577 connection.cc:535] client connection to 127.0.0.1:35019 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35019): Connection reset by peer (error 104)
2026-08-12T02:36:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1566 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786501898713699-14748-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 02:36:36 dist-test-slave-2w3w krb5kdc[6708](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 02:36:36 dist-test-slave-2w3w krb5kdc[6708](info): set up 2 sockets
Aug 12 02:36:36 dist-test-slave-2w3w krb5kdc[6708](info): commencing operation
krb5kdc: starting...
W20260812 02:36:37.837283 14748 mini_kdc.cc:121] Time spent starting KDC: real 1.099s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786501898713699-14748-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786501898713699-14748-0/krb5kdc/test-user.keytab.
Aug 12 02:36:37 dist-test-slave-2w3w krb5kdc[6708](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502197, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T02:36:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T02:36:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786501898713699-14748-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786501898713699-14748-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 02:36:37.982658 14748 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbep4M4/build/debug/bin/kudu
/tmp/dist-test-taskbep4M4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786501898713699-14748-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786501898713699-14748-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786501898713699-14748-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786501898713699-14748-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36117
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786501898713699-14748-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36117
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786501898713699-14748-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786501898713699-14748-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786501898713699-14748-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:36:38.103540 6721 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:36:38.103869 6721 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:36:38.103941 6721 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:36:38.108333 6721 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:36:38.108430 6721 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:36:38.108453 6721 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:36:38.108471 6721 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:36:38.108487 6721 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 02:36:38.114447 6721 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786501898713699-14748-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786501898713699-14748-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36117
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786501898713699-14748-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36117
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786501898713699-14748-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6721
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786501898713699-14748-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:36:38.115715 6721 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:36:38.116792 6721 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:36:38.126521 6727 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:36:38.126525 6728 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:36:38.126521 6730 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:36:38.126611 6721 server_base.cc:1034] running on GCE node
I20260812 02:36:38.127559 6721 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:36:38.128705 6721 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:36:38.129897 6721 hybrid_clock.cc:648] HybridClock initialized: now 1786502198129877 us; error 35 us; skew 500 ppm
Aug 12 02:36:38 dist-test-slave-2w3w krb5kdc[6708](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502198, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:36:38.136884 6721 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 02:36:38.138016 6721 webserver.cc:460] Webserver started at http://127.0.0.1:34991/ using document root <none> and password file <none>
I20260812 02:36:38.138667 6721 fs_manager.cc:359] Metadata directory not provided
I20260812 02:36:38.138772 6721 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:36:38.139092 6721 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:36:38.141306 6721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786501898713699-14748-0/minicluster-data/master-0/data/instance:
uuid: "e8a21a57e60347b8897f4f26c9c267c2"
format_stamp: "Formatted at 2026-08-12 02:36:38 on dist-test-slave-2w3w"
server_key: "9f6bf90f6b423371201fad59fb2be074"
server_key_iv: "bbc3b334ce0d4f01c188425fef33492b"
server_key_version: "encryptionkey@0"
I20260812 02:36:38.141924 6721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786501898713699-14748-0/minicluster-data/master-0/wal/instance:
uuid: "e8a21a57e60347b8897f4f26c9c267c2"
format_stamp: "Formatted at 2026-08-12 02:36:38 on dist-test-slave-2w3w"
server_key: "9f6bf90f6b423371201fad59fb2be074"
server_key_iv: "bbc3b334ce0d4f01c188425fef33492b"
server_key_version: "encryptionkey@0"
I20260812 02:36:38.145982 6721 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260812 02:36:38.148676 6738 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:36:38.150200 6721 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260812 02:36:38.150398 6721 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786501898713699-14748-0/minicluster-data/master-0/data,/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786501898713699-14748-0/minicluster-data/master-0/wal
uuid: "e8a21a57e60347b8897f4f26c9c267c2"
format_stamp: "Formatted at 2026-08-12 02:36:38 on dist-test-slave-2w3w"
server_key: "9f6bf90f6b423371201fad59fb2be074"
server_key_iv: "bbc3b334ce0d4f01c188425fef33492b"
server_key_version: "encryptionkey@0"
I20260812 02:36:38.150539 6721 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786501898713699-14748-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786501898713699-14748-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786501898713699-14748-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:36:38.167001 6721 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:36:38.170370 6721 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:36:38.170625 6721 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:36:38.178376 6721 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36117
I20260812 02:36:38.178404 6794 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36117 every 8 connection(s)
I20260812 02:36:38.179458 6721 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786501898713699-14748-0/minicluster-data/master-0/data/info.pb
I20260812 02:36:38.181495 14748 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbep4M4/build/debug/bin/kudu as pid 6721
I20260812 02:36:38.181649 14748 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786501898713699-14748-0/minicluster-data/master-0/wal/instance
I20260812 02:36:38.182024 14748 external_mini_cluster.cc:1428] Setting key b541d3254168195b0a358773d101ca5e
I20260812 02:36:38.182631 6795 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:36:38.188925 6795 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8a21a57e60347b8897f4f26c9c267c2: Bootstrap starting.
I20260812 02:36:38.191334 6795 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e8a21a57e60347b8897f4f26c9c267c2: Neither blocks nor log segments found. Creating new log.
I20260812 02:36:38.191358 6798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:38.183588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.84:44387 (local address 127.0.0.1:36117)
0812 02:36:38.184062 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0812 02:36:38.184077 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0812 02:36:38.184114 (+ 37us) server_negotiation.cc:407] Connection header received
0812 02:36:38.184784 (+ 670us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:36:38.184801 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:36:38.185136 (+ 335us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:36:38.185477 (+ 341us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:36:38.185902 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:38.186902 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:38.187897 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:38.188391 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:38.188581 (+ 190us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:36:38.189553 (+ 972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:36:38.189558 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:36:38.189570 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:36:38.189967 (+ 397us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:36:38.190024 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:36:38.190048 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:36:38.190200 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0812 02:36:38.190387 (+ 187us) server_negotiation.cc:299] Negotiation successful
0812 02:36:38.190544 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":156,"threads_started":1}
I20260812 02:36:38.192106 6795 log.cc:824] T 00000000000000000000000000000000 P e8a21a57e60347b8897f4f26c9c267c2: Log is configured to *not* fsync() on all Append() calls
I20260812 02:36:38.194379 6795 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8a21a57e60347b8897f4f26c9c267c2: No bootstrap required, opened a new log
I20260812 02:36:38.194568 14748 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 02:36:38.199226 6795 raft_consensus.cc:348] T 00000000000000000000000000000000 P e8a21a57e60347b8897f4f26c9c267c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8a21a57e60347b8897f4f26c9c267c2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36117 } }
I20260812 02:36:38.199545 6795 raft_consensus.cc:374] T 00000000000000000000000000000000 P e8a21a57e60347b8897f4f26c9c267c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:36:38.199612 6795 raft_consensus.cc:729] T 00000000000000000000000000000000 P e8a21a57e60347b8897f4f26c9c267c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8a21a57e60347b8897f4f26c9c267c2, State: Initialized, Role: FOLLOWER
I20260812 02:36:38.200134 6795 consensus_queue.cc:260] T 00000000000000000000000000000000 P e8a21a57e60347b8897f4f26c9c267c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8a21a57e60347b8897f4f26c9c267c2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36117 } }
I20260812 02:36:38.200320 6795 raft_consensus.cc:388] T 00000000000000000000000000000000 P e8a21a57e60347b8897f4f26c9c267c2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 02:36:38.200405 6795 raft_consensus.cc:482] T 00000000000000000000000000000000 P e8a21a57e60347b8897f4f26c9c267c2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 02:36:38.200515 6795 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e8a21a57e60347b8897f4f26c9c267c2 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:36:38.201570 6798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:38.196924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.84:35345 (local address 127.0.0.1:36117)
0812 02:36:38.197050 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0812 02:36:38.197054 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:36:38.197458 (+ 404us) server_negotiation.cc:407] Connection header received
0812 02:36:38.197571 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:36:38.197573 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:36:38.197616 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:36:38.197688 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:36:38.197997 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:38.198524 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:38.199648 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:38.199964 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:38.200055 (+ 91us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:36:38.201024 (+ 969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:36:38.201028 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:36:38.201031 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:36:38.201043 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:36:38.201071 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:36:38.201084 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:36:38.201182 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0812 02:36:38.201375 (+ 193us) server_negotiation.cc:299] Negotiation successful
0812 02:36:38.201416 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260812 02:36:38.201822 6795 raft_consensus.cc:504] T 00000000000000000000000000000000 P e8a21a57e60347b8897f4f26c9c267c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8a21a57e60347b8897f4f26c9c267c2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36117 } }
I20260812 02:36:38.202211 6795 leader_election.cc:302] T 00000000000000000000000000000000 P e8a21a57e60347b8897f4f26c9c267c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e8a21a57e60347b8897f4f26c9c267c2; no voters:
I20260812 02:36:38.202585 6795 leader_election.cc:288] T 00000000000000000000000000000000 P e8a21a57e60347b8897f4f26c9c267c2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 02:36:38.202718 6806 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e8a21a57e60347b8897f4f26c9c267c2 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:36:38.203116 6806 raft_consensus.cc:686] T 00000000000000000000000000000000 P e8a21a57e60347b8897f4f26c9c267c2 [term 1 LEADER]: Becoming Leader. State: Replica: e8a21a57e60347b8897f4f26c9c267c2, State: Running, Role: LEADER
I20260812 02:36:38.203521 6806 consensus_queue.cc:237] T 00000000000000000000000000000000 P e8a21a57e60347b8897f4f26c9c267c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8a21a57e60347b8897f4f26c9c267c2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36117 } }
I20260812 02:36:38.204036 6795 sys_catalog.cc:564] T 00000000000000000000000000000000 P e8a21a57e60347b8897f4f26c9c267c2 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:36:38.205291 6811 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8a21a57e60347b8897f4f26c9c267c2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e8a21a57e60347b8897f4f26c9c267c2. Latest consensus state: current_term: 1 leader_uuid: "e8a21a57e60347b8897f4f26c9c267c2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8a21a57e60347b8897f4f26c9c267c2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36117 } } }
I20260812 02:36:38.205407 6811 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8a21a57e60347b8897f4f26c9c267c2 [sys.catalog]: This master's current role is: LEADER
I20260812 02:36:38.205775 6807 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8a21a57e60347b8897f4f26c9c267c2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e8a21a57e60347b8897f4f26c9c267c2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8a21a57e60347b8897f4f26c9c267c2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36117 } } }
I20260812 02:36:38.205886 6807 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8a21a57e60347b8897f4f26c9c267c2 [sys.catalog]: This master's current role is: LEADER
I20260812 02:36:38.205960 6821 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 02:36:38.209738 6821 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 02:36:38.216009 6821 catalog_manager.cc:1347] Generated new cluster ID: 93199f72211a41069a1fd26cfc80badb
I20260812 02:36:38.216107 6821 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 02:36:38.250360 6821 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 02:36:38.251843 6821 catalog_manager.cc:1504] Loading token signing keys...
I20260812 02:36:38.259999 6821 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e8a21a57e60347b8897f4f26c9c267c2: Generated new TSK 0
I20260812 02:36:38.260911 6821 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 02:36:38.269757 14748 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbep4M4/build/debug/bin/kudu with pid 6721
W20260812 02:36:38.279129 6706 connection.cc:535] client connection to 127.0.0.1:36117 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36117): Connection reset by peer (error 104)
2026-08-12T02:36:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1556 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786501898713699-14748-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 02:36:38 dist-test-slave-2w3w krb5kdc[6833](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 02:36:38 dist-test-slave-2w3w krb5kdc[6833](info): set up 2 sockets
Aug 12 02:36:38 dist-test-slave-2w3w krb5kdc[6833](info): commencing operation
krb5kdc: starting...
W20260812 02:36:39.394309 14748 mini_kdc.cc:121] Time spent starting KDC: real 1.100s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786501898713699-14748-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786501898713699-14748-0/krb5kdc/test-user.keytab.
Aug 12 02:36:39 dist-test-slave-2w3w krb5kdc[6833](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502199, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T02:36:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T02:36:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786501898713699-14748-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786501898713699-14748-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 02:36:39.537917 14748 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbep4M4/build/debug/bin/kudu
/tmp/dist-test-taskbep4M4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786501898713699-14748-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786501898713699-14748-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786501898713699-14748-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786501898713699-14748-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46445
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786501898713699-14748-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46445
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786501898713699-14748-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786501898713699-14748-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786501898713699-14748-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:36:39.655531 6847 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:36:39.655846 6847 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:36:39.655949 6847 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:36:39.660086 6847 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:36:39.660162 6847 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:36:39.660183 6847 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:36:39.660200 6847 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:36:39.660260 6847 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 02:36:39.665787 6847 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786501898713699-14748-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786501898713699-14748-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46445
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786501898713699-14748-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46445
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786501898713699-14748-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6847
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786501898713699-14748-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:36:39.666986 6847 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:36:39.668061 6847 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:36:39.677567 6852 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:36:39.677613 6855 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:36:39.677819 6847 server_base.cc:1034] running on GCE node
W20260812 02:36:39.677567 6853 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:36:39.678562 6847 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:36:39.679844 6847 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:36:39.681090 6847 hybrid_clock.cc:648] HybridClock initialized: now 1786502199681061 us; error 46 us; skew 500 ppm
Aug 12 02:36:39 dist-test-slave-2w3w krb5kdc[6833](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502199, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:36:39.684340 6847 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 02:36:39.685527 6847 webserver.cc:460] Webserver started at http://127.0.0.1:39091/ using document root <none> and password file <none>
I20260812 02:36:39.686151 6847 fs_manager.cc:359] Metadata directory not provided
I20260812 02:36:39.686251 6847 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:36:39.686511 6847 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:36:39.688813 6847 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786501898713699-14748-0/minicluster-data/master-0/data/instance:
uuid: "eaa98674314445cb8c9a34f4dbe32bb6"
format_stamp: "Formatted at 2026-08-12 02:36:39 on dist-test-slave-2w3w"
server_key: "105a08cfb0808e77e3641223096101f2"
server_key_iv: "6790d9605e729c7e290a6f3af876fd24"
server_key_version: "encryptionkey@0"
I20260812 02:36:39.689491 6847 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786501898713699-14748-0/minicluster-data/master-0/wal/instance:
uuid: "eaa98674314445cb8c9a34f4dbe32bb6"
format_stamp: "Formatted at 2026-08-12 02:36:39 on dist-test-slave-2w3w"
server_key: "105a08cfb0808e77e3641223096101f2"
server_key_iv: "6790d9605e729c7e290a6f3af876fd24"
server_key_version: "encryptionkey@0"
I20260812 02:36:39.693832 6847 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.005s
I20260812 02:36:39.696902 6862 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:36:39.698355 6847 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260812 02:36:39.698534 6847 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786501898713699-14748-0/minicluster-data/master-0/data,/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786501898713699-14748-0/minicluster-data/master-0/wal
uuid: "eaa98674314445cb8c9a34f4dbe32bb6"
format_stamp: "Formatted at 2026-08-12 02:36:39 on dist-test-slave-2w3w"
server_key: "105a08cfb0808e77e3641223096101f2"
server_key_iv: "6790d9605e729c7e290a6f3af876fd24"
server_key_version: "encryptionkey@0"
I20260812 02:36:39.698673 6847 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786501898713699-14748-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786501898713699-14748-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786501898713699-14748-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:36:39.703907 6847 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:36:39.707576 6847 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:36:39.707763 6847 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:36:39.716133 6847 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46445
I20260812 02:36:39.716219 6920 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46445 every 8 connection(s)
I20260812 02:36:39.717259 6847 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786501898713699-14748-0/minicluster-data/master-0/data/info.pb
I20260812 02:36:39.720395 6921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:36:39.725931 14748 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbep4M4/build/debug/bin/kudu as pid 6847
I20260812 02:36:39.726058 14748 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786501898713699-14748-0/minicluster-data/master-0/wal/instance
I20260812 02:36:39.726378 14748 external_mini_cluster.cc:1428] Setting key 3a7022e59aaaa45dc94e3809234b2bd8
I20260812 02:36:39.726471 6921 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eaa98674314445cb8c9a34f4dbe32bb6: Bootstrap starting.
I20260812 02:36:39.729387 6921 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eaa98674314445cb8c9a34f4dbe32bb6: Neither blocks nor log segments found. Creating new log.
I20260812 02:36:39.730288 6921 log.cc:824] T 00000000000000000000000000000000 P eaa98674314445cb8c9a34f4dbe32bb6: Log is configured to *not* fsync() on all Append() calls
I20260812 02:36:39.732421 6925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:39.727887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.84:53693 (local address 127.0.0.1:46445)
0812 02:36:39.728401 (+ 514us) server_negotiation.cc:206] Beginning negotiation
0812 02:36:39.728420 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0812 02:36:39.728463 (+ 43us) server_negotiation.cc:407] Connection header received
0812 02:36:39.729341 (+ 878us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:36:39.729368 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:36:39.729747 (+ 379us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:36:39.730119 (+ 372us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:36:39.730926 (+ 807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:36:39.730930 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:36:39.730941 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:36:39.730951 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:36:39.731087 (+ 136us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:36:39.731114 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:36:39.731205 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0812 02:36:39.731379 (+ 174us) server_negotiation.cc:299] Negotiation successful
0812 02:36:39.731542 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":149,"threads_started":1}
I20260812 02:36:39.732800 6921 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eaa98674314445cb8c9a34f4dbe32bb6: No bootstrap required, opened a new log
I20260812 02:36:39.735392 14748 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 02:36:39.737059 6921 raft_consensus.cc:348] T 00000000000000000000000000000000 P eaa98674314445cb8c9a34f4dbe32bb6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eaa98674314445cb8c9a34f4dbe32bb6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46445 } }
I20260812 02:36:39.737326 6921 raft_consensus.cc:374] T 00000000000000000000000000000000 P eaa98674314445cb8c9a34f4dbe32bb6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:36:39.737450 6921 raft_consensus.cc:729] T 00000000000000000000000000000000 P eaa98674314445cb8c9a34f4dbe32bb6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaa98674314445cb8c9a34f4dbe32bb6, State: Initialized, Role: FOLLOWER
I20260812 02:36:39.737996 6921 consensus_queue.cc:260] T 00000000000000000000000000000000 P eaa98674314445cb8c9a34f4dbe32bb6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eaa98674314445cb8c9a34f4dbe32bb6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46445 } }
I20260812 02:36:39.738157 6921 raft_consensus.cc:388] T 00000000000000000000000000000000 P eaa98674314445cb8c9a34f4dbe32bb6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 02:36:39.738247 6921 raft_consensus.cc:482] T 00000000000000000000000000000000 P eaa98674314445cb8c9a34f4dbe32bb6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 02:36:39.738350 6921 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eaa98674314445cb8c9a34f4dbe32bb6 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:36:39.739207 6925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:39.737464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.84:41979 (local address 127.0.0.1:46445)
0812 02:36:39.737603 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0812 02:36:39.737608 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:36:39.737776 (+ 168us) server_negotiation.cc:407] Connection header received
0812 02:36:39.737921 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:36:39.737924 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:36:39.737966 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:36:39.738049 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:36:39.738766 (+ 717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:36:39.738770 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:36:39.738772 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:36:39.738784 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:36:39.738820 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:36:39.738832 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:36:39.738929 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0812 02:36:39.739039 (+ 110us) server_negotiation.cc:299] Negotiation successful
0812 02:36:39.739083 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260812 02:36:39.739619 6921 raft_consensus.cc:504] T 00000000000000000000000000000000 P eaa98674314445cb8c9a34f4dbe32bb6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eaa98674314445cb8c9a34f4dbe32bb6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46445 } }
I20260812 02:36:39.740015 6921 leader_election.cc:302] T 00000000000000000000000000000000 P eaa98674314445cb8c9a34f4dbe32bb6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eaa98674314445cb8c9a34f4dbe32bb6; no voters:
I20260812 02:36:39.740382 6921 leader_election.cc:288] T 00000000000000000000000000000000 P eaa98674314445cb8c9a34f4dbe32bb6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 02:36:39.740495 6933 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eaa98674314445cb8c9a34f4dbe32bb6 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:36:39.740744 6933 raft_consensus.cc:686] T 00000000000000000000000000000000 P eaa98674314445cb8c9a34f4dbe32bb6 [term 1 LEADER]: Becoming Leader. State: Replica: eaa98674314445cb8c9a34f4dbe32bb6, State: Running, Role: LEADER
I20260812 02:36:39.741108 6933 consensus_queue.cc:237] T 00000000000000000000000000000000 P eaa98674314445cb8c9a34f4dbe32bb6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eaa98674314445cb8c9a34f4dbe32bb6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46445 } }
I20260812 02:36:39.741683 6921 sys_catalog.cc:564] T 00000000000000000000000000000000 P eaa98674314445cb8c9a34f4dbe32bb6 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:36:39.742774 6935 sys_catalog.cc:455] T 00000000000000000000000000000000 P eaa98674314445cb8c9a34f4dbe32bb6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader eaa98674314445cb8c9a34f4dbe32bb6. Latest consensus state: current_term: 1 leader_uuid: "eaa98674314445cb8c9a34f4dbe32bb6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eaa98674314445cb8c9a34f4dbe32bb6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46445 } } }
I20260812 02:36:39.742913 6935 sys_catalog.cc:458] T 00000000000000000000000000000000 P eaa98674314445cb8c9a34f4dbe32bb6 [sys.catalog]: This master's current role is: LEADER
I20260812 02:36:39.743079 6934 sys_catalog.cc:455] T 00000000000000000000000000000000 P eaa98674314445cb8c9a34f4dbe32bb6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eaa98674314445cb8c9a34f4dbe32bb6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eaa98674314445cb8c9a34f4dbe32bb6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46445 } } }
I20260812 02:36:39.743243 6934 sys_catalog.cc:458] T 00000000000000000000000000000000 P eaa98674314445cb8c9a34f4dbe32bb6 [sys.catalog]: This master's current role is: LEADER
I20260812 02:36:39.744340 6943 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 02:36:39.748077 6943 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 02:36:39.754671 6943 catalog_manager.cc:1347] Generated new cluster ID: 27c3d1e29e6143e1bbc73963ce67b234
I20260812 02:36:39.754793 6943 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 02:36:39.776588 6943 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 02:36:39.776928 6943 catalog_manager.cc:1504] Loading token signing keys...
I20260812 02:36:39.781361 6943 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eaa98674314445cb8c9a34f4dbe32bb6: Generated new TSK 0
I20260812 02:36:39.782065 6943 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 02:36:39.787482 14748 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbep4M4/build/debug/bin/kudu with pid 6847
2026-08-12T02:36:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1518 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786501898713699-14748-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 02:36:39 dist-test-slave-2w3w krb5kdc[6957](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 02:36:39 dist-test-slave-2w3w krb5kdc[6957](info): set up 2 sockets
Aug 12 02:36:39 dist-test-slave-2w3w krb5kdc[6957](info): commencing operation
krb5kdc: starting...
W20260812 02:36:40.928719 14748 mini_kdc.cc:121] Time spent starting KDC: real 1.116s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786501898713699-14748-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786501898713699-14748-0/krb5kdc/test-user.keytab.
Aug 12 02:36:41 dist-test-slave-2w3w krb5kdc[6957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502201, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T02:36:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T02:36:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.103.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.103.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.103.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.62.keytab.
Entry for principal kudu/127.14.103.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.62.keytab.
I20260812 02:36:41.075614 14748 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbep4M4/build/debug/bin/kudu
/tmp/dist-test-taskbep4M4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786501898713699-14748-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786501898713699-14748-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786501898713699-14748-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786501898713699-14748-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.103.62:35177
--webserver_interface=127.14.103.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.62.keytab
--principal=kudu/127.14.103.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.103.20:39229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.103.62:35177
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786501898713699-14748-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786501898713699-14748-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786501898713699-14748-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:36:41.195119 6973 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:36:41.195490 6973 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:36:41.195573 6973 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:36:41.199878 6973 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:36:41.200001 6973 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:36:41.200039 6973 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:36:41.200083 6973 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:36:41.200120 6973 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 02:36:41.205969 6973 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.103.20:39229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786501898713699-14748-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786501898713699-14748-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.103.62:35177
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.62.keytab
--principal=kudu/127.14.103.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.103.62:35177
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786501898713699-14748-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.103.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6973
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786501898713699-14748-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:36:41.207278 6973 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:36:41.208343 6973 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:36:41.217718 6981 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:36:41.217732 6978 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:36:41.217738 6979 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:36:41.217953 6973 server_base.cc:1034] running on GCE node
I20260812 02:36:41.218766 6973 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:36:41.219949 6973 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:36:41.221159 6973 hybrid_clock.cc:648] HybridClock initialized: now 1786502201221113 us; error 53 us; skew 500 ppm
Aug 12 02:36:41 dist-test-slave-2w3w krb5kdc[6957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502201, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:36:41.224314 6973 init.cc:377] Logged in from keytab as kudu/127.14.103.62@KRBTEST.COM (short username kudu)
I20260812 02:36:41.225262 6973 webserver.cc:460] Webserver started at http://127.14.103.62:46851/ using document root <none> and password file <none>
I20260812 02:36:41.225802 6973 fs_manager.cc:359] Metadata directory not provided
I20260812 02:36:41.225872 6973 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:36:41.226049 6973 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:36:41.228310 6973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786501898713699-14748-0/minicluster-data/master-0/data/instance:
uuid: "5709c33f98b0478eae349806641376db"
format_stamp: "Formatted at 2026-08-12 02:36:41 on dist-test-slave-2w3w"
server_key: "eb6c782d4d3c10cea16f9155afce545d"
server_key_iv: "226b7ff77857c467e39c9833f95ee1ba"
server_key_version: "encryptionkey@0"
I20260812 02:36:41.228911 6973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786501898713699-14748-0/minicluster-data/master-0/wal/instance:
uuid: "5709c33f98b0478eae349806641376db"
format_stamp: "Formatted at 2026-08-12 02:36:41 on dist-test-slave-2w3w"
server_key: "eb6c782d4d3c10cea16f9155afce545d"
server_key_iv: "226b7ff77857c467e39c9833f95ee1ba"
server_key_version: "encryptionkey@0"
I20260812 02:36:41.233291 6973 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260812 02:36:41.236050 6990 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:36:41.237468 6973 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 02:36:41.237593 6973 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786501898713699-14748-0/minicluster-data/master-0/data,/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786501898713699-14748-0/minicluster-data/master-0/wal
uuid: "5709c33f98b0478eae349806641376db"
format_stamp: "Formatted at 2026-08-12 02:36:41 on dist-test-slave-2w3w"
server_key: "eb6c782d4d3c10cea16f9155afce545d"
server_key_iv: "226b7ff77857c467e39c9833f95ee1ba"
server_key_version: "encryptionkey@0"
I20260812 02:36:41.237749 6973 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786501898713699-14748-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786501898713699-14748-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786501898713699-14748-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:36:41.268321 6973 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:36:41.272169 6973 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:36:41.272423 6973 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:36:41.281637 6973 rpc_server.cc:307] RPC server started. Bound to: 127.14.103.62:35177
I20260812 02:36:41.281646 7046 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.103.62:35177 every 8 connection(s)
I20260812 02:36:41.282737 6973 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786501898713699-14748-0/minicluster-data/master-0/data/info.pb
I20260812 02:36:41.284581 14748 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbep4M4/build/debug/bin/kudu as pid 6973
I20260812 02:36:41.284736 14748 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786501898713699-14748-0/minicluster-data/master-0/wal/instance
I20260812 02:36:41.285079 14748 external_mini_cluster.cc:1428] Setting key c146520767163ae48b45bb7f85e47e77
I20260812 02:36:41.286269 7047 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:36:41.293298 7047 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5709c33f98b0478eae349806641376db: Bootstrap starting.
Aug 12 02:36:41 dist-test-slave-2w3w krb5kdc[6957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502201, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.103.62@KRBTEST.COM
I20260812 02:36:41.296139 7047 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5709c33f98b0478eae349806641376db: Neither blocks nor log segments found. Creating new log.
I20260812 02:36:41.297137 7047 log.cc:824] T 00000000000000000000000000000000 P 5709c33f98b0478eae349806641376db: Log is configured to *not* fsync() on all Append() calls
I20260812 02:36:41.299780 7047 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5709c33f98b0478eae349806641376db: No bootstrap required, opened a new log
I20260812 02:36:41.301082 7050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:41.286813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39715 (local address 127.14.103.62:35177)
0812 02:36:41.287302 (+ 489us) server_negotiation.cc:206] Beginning negotiation
0812 02:36:41.287317 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0812 02:36:41.287354 (+ 37us) server_negotiation.cc:407] Connection header received
0812 02:36:41.288018 (+ 664us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:36:41.288034 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:36:41.288389 (+ 355us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:36:41.288681 (+ 292us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:36:41.289704 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:41.290771 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:41.292007 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:41.292488 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:41.292617 (+ 129us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:36:41.295567 (+ 2950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:36:41.295574 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:36:41.295590 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:36:41.295625 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:36:41.297718 (+ 2093us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:36:41.298359 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:36:41.298365 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:36:41.298371 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:36:41.298507 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:36:41.298830 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:36:41.298833 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:36:41.298835 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:36:41.299391 (+ 556us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:36:41.299603 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0812 02:36:41.299886 (+ 283us) server_negotiation.cc:299] Negotiation successful
0812 02:36:41.300124 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":187,"threads_started":1}
I20260812 02:36:41.303820 7047 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5709c33f98b0478eae349806641376db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5709c33f98b0478eae349806641376db" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 35177 } }
I20260812 02:36:41.304080 7047 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5709c33f98b0478eae349806641376db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:36:41.304155 7047 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5709c33f98b0478eae349806641376db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5709c33f98b0478eae349806641376db, State: Initialized, Role: FOLLOWER
I20260812 02:36:41.304607 7047 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5709c33f98b0478eae349806641376db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5709c33f98b0478eae349806641376db" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 35177 } }
I20260812 02:36:41.304761 7047 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5709c33f98b0478eae349806641376db [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 02:36:41.304842 7047 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5709c33f98b0478eae349806641376db [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 02:36:41.304875 14748 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 02:36:41.304991 7047 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5709c33f98b0478eae349806641376db [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:36:41.306051 7047 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5709c33f98b0478eae349806641376db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5709c33f98b0478eae349806641376db" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 35177 } }
I20260812 02:36:41.306421 7047 leader_election.cc:302] T 00000000000000000000000000000000 P 5709c33f98b0478eae349806641376db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5709c33f98b0478eae349806641376db; no voters:
I20260812 02:36:41.306782 7047 leader_election.cc:288] T 00000000000000000000000000000000 P 5709c33f98b0478eae349806641376db [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 02:36:41.306864 7057 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5709c33f98b0478eae349806641376db [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:36:41.307130 7057 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5709c33f98b0478eae349806641376db [term 1 LEADER]: Becoming Leader. State: Replica: 5709c33f98b0478eae349806641376db, State: Running, Role: LEADER
I20260812 02:36:41.307515 7057 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5709c33f98b0478eae349806641376db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5709c33f98b0478eae349806641376db" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 35177 } }
I20260812 02:36:41.308255 7047 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5709c33f98b0478eae349806641376db [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:36:41.308919 7057 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5709c33f98b0478eae349806641376db [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5709c33f98b0478eae349806641376db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5709c33f98b0478eae349806641376db" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 35177 } } }
I20260812 02:36:41.309127 7057 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5709c33f98b0478eae349806641376db [sys.catalog]: This master's current role is: LEADER
I20260812 02:36:41.309401 7057 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5709c33f98b0478eae349806641376db [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5709c33f98b0478eae349806641376db. Latest consensus state: current_term: 1 leader_uuid: "5709c33f98b0478eae349806641376db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5709c33f98b0478eae349806641376db" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 35177 } } }
I20260812 02:36:41.309559 7057 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5709c33f98b0478eae349806641376db [sys.catalog]: This master's current role is: LEADER
I20260812 02:36:41.309870 7066 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 02:36:41.313329 7066 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 02:36:41.316562 7050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:41.307139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35221 (local address 127.14.103.62:35177)
0812 02:36:41.307280 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0812 02:36:41.307286 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 02:36:41.307505 (+ 219us) server_negotiation.cc:407] Connection header received
0812 02:36:41.307653 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:36:41.307656 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:36:41.307710 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:36:41.307802 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:36:41.308598 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:41.309421 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:41.310567 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:41.310881 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:41.311039 (+ 158us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:36:41.312816 (+ 1777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:36:41.312820 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:36:41.312823 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:36:41.312850 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:36:41.314468 (+ 1618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:36:41.315164 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:36:41.315167 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:36:41.315169 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:36:41.315257 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:36:41.315634 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:36:41.315637 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:36:41.315639 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:36:41.315837 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:36:41.315995 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0812 02:36:41.316299 (+ 304us) server_negotiation.cc:299] Negotiation successful
0812 02:36:41.316398 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260812 02:36:41.320058 7066 catalog_manager.cc:1347] Generated new cluster ID: 7437b801f09f458caea09f8f7c111cc6
I20260812 02:36:41.320142 7066 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 02:36:41.326057 7066 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 02:36:41.326943 7066 catalog_manager.cc:1504] Loading token signing keys...
I20260812 02:36:41.334995 7066 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5709c33f98b0478eae349806641376db: Generated new TSK 0
I20260812 02:36:41.335736 7066 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 02:36:41.341236 14748 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbep4M4/build/debug/bin/kudu with pid 6973
W20260812 02:36:41.348894 6955 connection.cc:535] client connection to 127.14.103.62:35177 recv error: Network error: failed to read from TLS socket (remote: 127.14.103.62:35177): Connection reset by peer (error 104)
2026-08-12T02:36:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1550 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786501898713699-14748-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 02:36:41 dist-test-slave-2w3w krb5kdc[7081](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 02:36:41 dist-test-slave-2w3w krb5kdc[7081](info): set up 2 sockets
Aug 12 02:36:41 dist-test-slave-2w3w krb5kdc[7081](info): commencing operation
krb5kdc: starting...
W20260812 02:36:42.542610 14748 mini_kdc.cc:121] Time spent starting KDC: real 1.179s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786501898713699-14748-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786501898713699-14748-0/krb5kdc/test-user.keytab.
Aug 12 02:36:42 dist-test-slave-2w3w krb5kdc[7081](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502202, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T02:36:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T02:36:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.103.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.103.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.103.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.62.keytab.
Entry for principal kudu/127.14.103.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.62.keytab.
I20260812 02:36:42.683578 14748 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbep4M4/build/debug/bin/kudu
/tmp/dist-test-taskbep4M4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786501898713699-14748-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786501898713699-14748-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786501898713699-14748-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786501898713699-14748-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.103.62:42337
--webserver_interface=127.14.103.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.62.keytab
--principal=kudu/127.14.103.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.103.20:34059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.103.62:42337
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786501898713699-14748-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786501898713699-14748-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786501898713699-14748-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:36:42.798544 7097 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:36:42.798929 7097 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:36:42.799075 7097 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:36:42.803296 7097 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:36:42.803413 7097 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:36:42.803454 7097 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:36:42.803493 7097 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:36:42.803527 7097 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 02:36:42.809185 7097 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.103.20:34059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786501898713699-14748-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786501898713699-14748-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.103.62:42337
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.62.keytab
--principal=kudu/127.14.103.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.103.62:42337
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786501898713699-14748-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.103.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7097
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786501898713699-14748-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:36:42.810429 7097 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:36:42.811569 7097 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:36:42.820742 7102 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:36:42.820761 7105 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:36:42.820742 7103 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:36:42.820993 7097 server_base.cc:1034] running on GCE node
I20260812 02:36:42.821807 7097 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:36:42.823150 7097 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:36:42.824343 7097 hybrid_clock.cc:648] HybridClock initialized: now 1786502202824315 us; error 48 us; skew 500 ppm
Aug 12 02:36:42 dist-test-slave-2w3w krb5kdc[7081](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502202, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:36:42.827556 7097 init.cc:377] Logged in from keytab as kudu/127.14.103.62@KRBTEST.COM (short username kudu)
I20260812 02:36:42.828742 7097 webserver.cc:460] Webserver started at http://127.14.103.62:39569/ using document root <none> and password file <none>
I20260812 02:36:42.829383 7097 fs_manager.cc:359] Metadata directory not provided
I20260812 02:36:42.829459 7097 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:36:42.829695 7097 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:36:42.832044 7097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786501898713699-14748-0/minicluster-data/master-0/data/instance:
uuid: "c40862f6d24540a9afc27506a5e3c3cc"
format_stamp: "Formatted at 2026-08-12 02:36:42 on dist-test-slave-2w3w"
server_key: "69fad4f29846ab5b2735c34982bd8f7a"
server_key_iv: "16e5fcc68400dda7e204e6a35990c1db"
server_key_version: "encryptionkey@0"
I20260812 02:36:42.832652 7097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786501898713699-14748-0/minicluster-data/master-0/wal/instance:
uuid: "c40862f6d24540a9afc27506a5e3c3cc"
format_stamp: "Formatted at 2026-08-12 02:36:42 on dist-test-slave-2w3w"
server_key: "69fad4f29846ab5b2735c34982bd8f7a"
server_key_iv: "16e5fcc68400dda7e204e6a35990c1db"
server_key_version: "encryptionkey@0"
I20260812 02:36:42.836755 7097 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260812 02:36:42.839509 7113 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:36:42.840912 7097 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 02:36:42.841099 7097 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786501898713699-14748-0/minicluster-data/master-0/data,/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786501898713699-14748-0/minicluster-data/master-0/wal
uuid: "c40862f6d24540a9afc27506a5e3c3cc"
format_stamp: "Formatted at 2026-08-12 02:36:42 on dist-test-slave-2w3w"
server_key: "69fad4f29846ab5b2735c34982bd8f7a"
server_key_iv: "16e5fcc68400dda7e204e6a35990c1db"
server_key_version: "encryptionkey@0"
I20260812 02:36:42.841234 7097 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786501898713699-14748-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786501898713699-14748-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786501898713699-14748-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:36:42.861675 7097 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:36:42.865393 7097 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:36:42.865643 7097 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:36:42.873673 7097 rpc_server.cc:307] RPC server started. Bound to: 127.14.103.62:42337
I20260812 02:36:42.873690 7175 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.103.62:42337 every 8 connection(s)
I20260812 02:36:42.874722 7097 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786501898713699-14748-0/minicluster-data/master-0/data/info.pb
I20260812 02:36:42.877581 7176 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:36:42.882244 14748 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbep4M4/build/debug/bin/kudu as pid 7097
I20260812 02:36:42.882397 14748 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786501898713699-14748-0/minicluster-data/master-0/wal/instance
I20260812 02:36:42.882705 14748 external_mini_cluster.cc:1428] Setting key 43d0fed8b26c81710d1fe963a897a550
I20260812 02:36:42.883770 7176 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c40862f6d24540a9afc27506a5e3c3cc: Bootstrap starting.
I20260812 02:36:42.886039 7176 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c40862f6d24540a9afc27506a5e3c3cc: Neither blocks nor log segments found. Creating new log.
I20260812 02:36:42.886811 7176 log.cc:824] T 00000000000000000000000000000000 P c40862f6d24540a9afc27506a5e3c3cc: Log is configured to *not* fsync() on all Append() calls
I20260812 02:36:42.888722 7176 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c40862f6d24540a9afc27506a5e3c3cc: No bootstrap required, opened a new log
I20260812 02:36:42.892037 7179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:42.884287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47249 (local address 127.14.103.62:42337)
0812 02:36:42.884724 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0812 02:36:42.884741 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0812 02:36:42.884779 (+ 38us) server_negotiation.cc:407] Connection header received
0812 02:36:42.885431 (+ 652us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:36:42.885448 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:36:42.885787 (+ 339us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:36:42.886138 (+ 351us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:36:42.886591 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:42.887694 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:42.888633 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:42.889175 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:42.889353 (+ 178us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:36:42.890315 (+ 962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:36:42.890320 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:36:42.890332 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:36:42.890722 (+ 390us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:36:42.890775 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:36:42.890809 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:36:42.890957 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0812 02:36:42.891149 (+ 192us) server_negotiation.cc:299] Negotiation successful
0812 02:36:42.891301 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":150,"threads_started":1}
I20260812 02:36:42.892508 7176 raft_consensus.cc:348] T 00000000000000000000000000000000 P c40862f6d24540a9afc27506a5e3c3cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c40862f6d24540a9afc27506a5e3c3cc" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 42337 } }
I20260812 02:36:42.892753 7176 raft_consensus.cc:374] T 00000000000000000000000000000000 P c40862f6d24540a9afc27506a5e3c3cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:36:42.892860 7176 raft_consensus.cc:729] T 00000000000000000000000000000000 P c40862f6d24540a9afc27506a5e3c3cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c40862f6d24540a9afc27506a5e3c3cc, State: Initialized, Role: FOLLOWER
I20260812 02:36:42.893347 7176 consensus_queue.cc:260] T 00000000000000000000000000000000 P c40862f6d24540a9afc27506a5e3c3cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c40862f6d24540a9afc27506a5e3c3cc" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 42337 } }
I20260812 02:36:42.893508 7176 raft_consensus.cc:388] T 00000000000000000000000000000000 P c40862f6d24540a9afc27506a5e3c3cc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 02:36:42.893595 7176 raft_consensus.cc:482] T 00000000000000000000000000000000 P c40862f6d24540a9afc27506a5e3c3cc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 02:36:42.893716 7176 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c40862f6d24540a9afc27506a5e3c3cc [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:36:42.894863 7176 raft_consensus.cc:504] T 00000000000000000000000000000000 P c40862f6d24540a9afc27506a5e3c3cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c40862f6d24540a9afc27506a5e3c3cc" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 42337 } }
I20260812 02:36:42.895102 14748 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 02:36:42.895305 7176 leader_election.cc:302] T 00000000000000000000000000000000 P c40862f6d24540a9afc27506a5e3c3cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c40862f6d24540a9afc27506a5e3c3cc; no voters:
I20260812 02:36:42.895618 7176 leader_election.cc:288] T 00000000000000000000000000000000 P c40862f6d24540a9afc27506a5e3c3cc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 02:36:42.895747 7181 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c40862f6d24540a9afc27506a5e3c3cc [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:36:42.896023 7181 raft_consensus.cc:686] T 00000000000000000000000000000000 P c40862f6d24540a9afc27506a5e3c3cc [term 1 LEADER]: Becoming Leader. State: Replica: c40862f6d24540a9afc27506a5e3c3cc, State: Running, Role: LEADER
I20260812 02:36:42.896414 7181 consensus_queue.cc:237] T 00000000000000000000000000000000 P c40862f6d24540a9afc27506a5e3c3cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c40862f6d24540a9afc27506a5e3c3cc" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 42337 } }
I20260812 02:36:42.896910 7176 sys_catalog.cc:564] T 00000000000000000000000000000000 P c40862f6d24540a9afc27506a5e3c3cc [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:36:42.897684 7181 sys_catalog.cc:455] T 00000000000000000000000000000000 P c40862f6d24540a9afc27506a5e3c3cc [sys.catalog]: SysCatalogTable state changed. Reason: New leader c40862f6d24540a9afc27506a5e3c3cc. Latest consensus state: current_term: 1 leader_uuid: "c40862f6d24540a9afc27506a5e3c3cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c40862f6d24540a9afc27506a5e3c3cc" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 42337 } } }
I20260812 02:36:42.897912 7181 sys_catalog.cc:458] T 00000000000000000000000000000000 P c40862f6d24540a9afc27506a5e3c3cc [sys.catalog]: This master's current role is: LEADER
I20260812 02:36:42.898140 7182 sys_catalog.cc:455] T 00000000000000000000000000000000 P c40862f6d24540a9afc27506a5e3c3cc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c40862f6d24540a9afc27506a5e3c3cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c40862f6d24540a9afc27506a5e3c3cc" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 42337 } } }
I20260812 02:36:42.898257 7182 sys_catalog.cc:458] T 00000000000000000000000000000000 P c40862f6d24540a9afc27506a5e3c3cc [sys.catalog]: This master's current role is: LEADER
I20260812 02:36:42.898368 7196 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 02:36:42.901659 7196 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 02:36:42.902830 7179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:42.897438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44277 (local address 127.14.103.62:42337)
0812 02:36:42.897541 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0812 02:36:42.897545 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:36:42.897878 (+ 333us) server_negotiation.cc:407] Connection header received
0812 02:36:42.898084 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:36:42.898091 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:36:42.898146 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:36:42.898205 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:36:42.898674 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:42.899499 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:42.900665 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:42.900984 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:42.901089 (+ 105us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:36:42.902168 (+ 1079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:36:42.902173 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:36:42.902177 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:36:42.902190 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:36:42.902217 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:36:42.902231 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:36:42.902314 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0812 02:36:42.902646 (+ 332us) server_negotiation.cc:299] Negotiation successful
0812 02:36:42.902681 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260812 02:36:42.911155 7196 catalog_manager.cc:1347] Generated new cluster ID: 2335b8dddd24453fbf25697bb0d331b1
I20260812 02:36:42.911257 7196 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 02:36:42.929286 7196 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 02:36:42.930335 7196 catalog_manager.cc:1504] Loading token signing keys...
I20260812 02:36:42.942124 7196 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c40862f6d24540a9afc27506a5e3c3cc: Generated new TSK 0
I20260812 02:36:42.942878 7196 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 02:36:42.954525 14748 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbep4M4/build/debug/bin/kudu with pid 7097
W20260812 02:36:42.962287 7079 connection.cc:535] client connection to 127.14.103.62:42337 recv error: Network error: failed to read from TLS socket (remote: 127.14.103.62:42337): Connection reset by peer (error 104)
2026-08-12T02:36:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1613 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786501898713699-14748-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 02:36:43 dist-test-slave-2w3w krb5kdc[7209](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 02:36:43 dist-test-slave-2w3w krb5kdc[7209](info): set up 2 sockets
Aug 12 02:36:43 dist-test-slave-2w3w krb5kdc[7209](info): commencing operation
krb5kdc: starting...
W20260812 02:36:44.070377 14748 mini_kdc.cc:121] Time spent starting KDC: real 1.093s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786501898713699-14748-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786501898713699-14748-0/krb5kdc/test-user.keytab.
Aug 12 02:36:44 dist-test-slave-2w3w krb5kdc[7209](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502204, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T02:36:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T02:36:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.103.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.103.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.103.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.62.keytab.
Entry for principal kudu/127.14.103.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.62.keytab.
I20260812 02:36:44.215709 14748 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbep4M4/build/debug/bin/kudu
/tmp/dist-test-taskbep4M4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786501898713699-14748-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786501898713699-14748-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786501898713699-14748-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786501898713699-14748-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.103.62:38287
--webserver_interface=127.14.103.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.62.keytab
--principal=kudu/127.14.103.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.103.20:40125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.103.62:38287
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786501898713699-14748-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786501898713699-14748-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786501898713699-14748-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:36:44.331179 7225 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:36:44.331552 7225 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:36:44.331638 7225 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:36:44.335951 7225 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:36:44.336063 7225 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:36:44.336103 7225 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:36:44.336166 7225 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:36:44.336207 7225 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 02:36:44.342038 7225 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.103.20:40125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786501898713699-14748-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786501898713699-14748-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.103.62:38287
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.62.keytab
--principal=kudu/127.14.103.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.103.62:38287
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786501898713699-14748-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.103.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7225
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786501898713699-14748-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:36:44.343362 7225 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:36:44.344456 7225 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:36:44.353592 7234 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:36:44.353627 7236 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:36:44.353655 7233 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:36:44.353755 7225 server_base.cc:1034] running on GCE node
I20260812 02:36:44.354569 7225 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:36:44.355896 7225 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:36:44.357136 7225 hybrid_clock.cc:648] HybridClock initialized: now 1786502204357094 us; error 47 us; skew 500 ppm
Aug 12 02:36:44 dist-test-slave-2w3w krb5kdc[7209](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502204, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:36:44.364531 7225 init.cc:377] Logged in from keytab as kudu/127.14.103.62@KRBTEST.COM (short username kudu)
I20260812 02:36:44.365581 7225 webserver.cc:460] Webserver started at http://127.14.103.62:45995/ using document root <none> and password file <none>
I20260812 02:36:44.366186 7225 fs_manager.cc:359] Metadata directory not provided
I20260812 02:36:44.366284 7225 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:36:44.366539 7225 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:36:44.368798 7225 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786501898713699-14748-0/minicluster-data/master-0/data/instance:
uuid: "da3187b886114fc0ac4dad46a77e89fe"
format_stamp: "Formatted at 2026-08-12 02:36:44 on dist-test-slave-2w3w"
server_key: "b706a7c96c5071e4137a2d47634393cc"
server_key_iv: "fb0f86ec4009fa32050d47f44433f3d6"
server_key_version: "encryptionkey@0"
I20260812 02:36:44.369410 7225 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786501898713699-14748-0/minicluster-data/master-0/wal/instance:
uuid: "da3187b886114fc0ac4dad46a77e89fe"
format_stamp: "Formatted at 2026-08-12 02:36:44 on dist-test-slave-2w3w"
server_key: "b706a7c96c5071e4137a2d47634393cc"
server_key_iv: "fb0f86ec4009fa32050d47f44433f3d6"
server_key_version: "encryptionkey@0"
I20260812 02:36:44.373513 7225 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260812 02:36:44.376108 7243 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:36:44.377460 7225 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 02:36:44.377614 7225 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786501898713699-14748-0/minicluster-data/master-0/data,/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786501898713699-14748-0/minicluster-data/master-0/wal
uuid: "da3187b886114fc0ac4dad46a77e89fe"
format_stamp: "Formatted at 2026-08-12 02:36:44 on dist-test-slave-2w3w"
server_key: "b706a7c96c5071e4137a2d47634393cc"
server_key_iv: "fb0f86ec4009fa32050d47f44433f3d6"
server_key_version: "encryptionkey@0"
I20260812 02:36:44.377737 7225 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786501898713699-14748-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786501898713699-14748-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786501898713699-14748-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:36:44.383001 7225 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:36:44.386387 7225 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:36:44.386548 7225 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:36:44.393811 7225 rpc_server.cc:307] RPC server started. Bound to: 127.14.103.62:38287
I20260812 02:36:44.393826 7302 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.103.62:38287 every 8 connection(s)
I20260812 02:36:44.394833 7225 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786501898713699-14748-0/minicluster-data/master-0/data/info.pb
I20260812 02:36:44.397785 7303 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:36:44.403543 7303 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da3187b886114fc0ac4dad46a77e89fe: Bootstrap starting.
I20260812 02:36:44.403806 14748 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbep4M4/build/debug/bin/kudu as pid 7225
I20260812 02:36:44.403985 14748 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786501898713699-14748-0/minicluster-data/master-0/wal/instance
I20260812 02:36:44.404309 14748 external_mini_cluster.cc:1428] Setting key 9d2c8de3467a5bce3950076d4969b9e6
I20260812 02:36:44.406237 7303 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P da3187b886114fc0ac4dad46a77e89fe: Neither blocks nor log segments found. Creating new log.
I20260812 02:36:44.407199 7303 log.cc:824] T 00000000000000000000000000000000 P da3187b886114fc0ac4dad46a77e89fe: Log is configured to *not* fsync() on all Append() calls
I20260812 02:36:44.409410 7303 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da3187b886114fc0ac4dad46a77e89fe: No bootstrap required, opened a new log
I20260812 02:36:44.410043 7307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:44.405684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39087 (local address 127.14.103.62:38287)
0812 02:36:44.406168 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0812 02:36:44.406184 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0812 02:36:44.406223 (+ 39us) server_negotiation.cc:407] Connection header received
0812 02:36:44.407039 (+ 816us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:36:44.407071 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:36:44.407430 (+ 359us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:36:44.407798 (+ 368us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:36:44.408706 (+ 908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:36:44.408711 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:36:44.408724 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:36:44.408736 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:36:44.408796 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:36:44.408818 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:36:44.408951 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0812 02:36:44.409063 (+ 112us) server_negotiation.cc:299] Negotiation successful
0812 02:36:44.409237 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":159,"threads_started":1}
I20260812 02:36:44.413007 14748 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 02:36:44.413717 7303 raft_consensus.cc:348] T 00000000000000000000000000000000 P da3187b886114fc0ac4dad46a77e89fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da3187b886114fc0ac4dad46a77e89fe" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 38287 } }
I20260812 02:36:44.413972 7303 raft_consensus.cc:374] T 00000000000000000000000000000000 P da3187b886114fc0ac4dad46a77e89fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:36:44.414062 7303 raft_consensus.cc:729] T 00000000000000000000000000000000 P da3187b886114fc0ac4dad46a77e89fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da3187b886114fc0ac4dad46a77e89fe, State: Initialized, Role: FOLLOWER
I20260812 02:36:44.414527 7303 consensus_queue.cc:260] T 00000000000000000000000000000000 P da3187b886114fc0ac4dad46a77e89fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da3187b886114fc0ac4dad46a77e89fe" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 38287 } }
I20260812 02:36:44.414681 7303 raft_consensus.cc:388] T 00000000000000000000000000000000 P da3187b886114fc0ac4dad46a77e89fe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 02:36:44.414783 7303 raft_consensus.cc:482] T 00000000000000000000000000000000 P da3187b886114fc0ac4dad46a77e89fe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 02:36:44.414884 7303 raft_consensus.cc:3037] T 00000000000000000000000000000000 P da3187b886114fc0ac4dad46a77e89fe [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:36:44.416291 7303 raft_consensus.cc:504] T 00000000000000000000000000000000 P da3187b886114fc0ac4dad46a77e89fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da3187b886114fc0ac4dad46a77e89fe" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 38287 } }
I20260812 02:36:44.416700 7303 leader_election.cc:302] T 00000000000000000000000000000000 P da3187b886114fc0ac4dad46a77e89fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: da3187b886114fc0ac4dad46a77e89fe; no voters:
I20260812 02:36:44.416909 7307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:44.415279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49785 (local address 127.14.103.62:38287)
0812 02:36:44.415415 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0812 02:36:44.415420 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:36:44.415570 (+ 150us) server_negotiation.cc:407] Connection header received
0812 02:36:44.415675 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:36:44.415678 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:36:44.415715 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:36:44.415793 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:36:44.416526 (+ 733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:36:44.416531 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:36:44.416533 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:36:44.416545 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:36:44.416576 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:36:44.416589 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:36:44.416680 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0812 02:36:44.416742 (+ 62us) server_negotiation.cc:299] Negotiation successful
0812 02:36:44.416780 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260812 02:36:44.417097 7303 leader_election.cc:288] T 00000000000000000000000000000000 P da3187b886114fc0ac4dad46a77e89fe [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 02:36:44.417188 7315 raft_consensus.cc:2781] T 00000000000000000000000000000000 P da3187b886114fc0ac4dad46a77e89fe [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:36:44.417408 7315 raft_consensus.cc:686] T 00000000000000000000000000000000 P da3187b886114fc0ac4dad46a77e89fe [term 1 LEADER]: Becoming Leader. State: Replica: da3187b886114fc0ac4dad46a77e89fe, State: Running, Role: LEADER
I20260812 02:36:44.417722 7315 consensus_queue.cc:237] T 00000000000000000000000000000000 P da3187b886114fc0ac4dad46a77e89fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da3187b886114fc0ac4dad46a77e89fe" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 38287 } }
I20260812 02:36:44.418386 7303 sys_catalog.cc:564] T 00000000000000000000000000000000 P da3187b886114fc0ac4dad46a77e89fe [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:36:44.419497 7317 sys_catalog.cc:455] T 00000000000000000000000000000000 P da3187b886114fc0ac4dad46a77e89fe [sys.catalog]: SysCatalogTable state changed. Reason: New leader da3187b886114fc0ac4dad46a77e89fe. Latest consensus state: current_term: 1 leader_uuid: "da3187b886114fc0ac4dad46a77e89fe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da3187b886114fc0ac4dad46a77e89fe" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 38287 } } }
I20260812 02:36:44.419646 7317 sys_catalog.cc:458] T 00000000000000000000000000000000 P da3187b886114fc0ac4dad46a77e89fe [sys.catalog]: This master's current role is: LEADER
I20260812 02:36:44.419948 7316 sys_catalog.cc:455] T 00000000000000000000000000000000 P da3187b886114fc0ac4dad46a77e89fe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "da3187b886114fc0ac4dad46a77e89fe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da3187b886114fc0ac4dad46a77e89fe" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 38287 } } }
I20260812 02:36:44.420032 7316 sys_catalog.cc:458] T 00000000000000000000000000000000 P da3187b886114fc0ac4dad46a77e89fe [sys.catalog]: This master's current role is: LEADER
I20260812 02:36:44.420066 7326 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 02:36:44.423486 7326 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 02:36:44.429631 7326 catalog_manager.cc:1347] Generated new cluster ID: 3966d2b266e94cc499bf7e35aca8ff14
I20260812 02:36:44.429731 7326 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 02:36:44.441442 7326 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 02:36:44.441809 7326 catalog_manager.cc:1504] Loading token signing keys...
I20260812 02:36:44.448098 7326 catalog_manager.cc:5868] T 00000000000000000000000000000000 P da3187b886114fc0ac4dad46a77e89fe: Generated new TSK 0
I20260812 02:36:44.448967 7326 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 02:36:44.459841 14748 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbep4M4/build/debug/bin/kudu with pid 7225
2026-08-12T02:36:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1506 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786501898713699-14748-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 02:36:44 dist-test-slave-2w3w krb5kdc[7339](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 02:36:44 dist-test-slave-2w3w krb5kdc[7339](info): set up 2 sockets
Aug 12 02:36:44 dist-test-slave-2w3w krb5kdc[7339](info): commencing operation
krb5kdc: starting...
W20260812 02:36:45.606213 14748 mini_kdc.cc:121] Time spent starting KDC: real 1.122s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786501898713699-14748-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786501898713699-14748-0/krb5kdc/test-user.keytab.
Aug 12 02:36:45 dist-test-slave-2w3w krb5kdc[7339](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502205, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T02:36:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T02:36:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.103.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.103.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.103.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.62.keytab.
Entry for principal kudu/127.14.103.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.62.keytab.
I20260812 02:36:45.755697 14748 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbep4M4/build/debug/bin/kudu
/tmp/dist-test-taskbep4M4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786501898713699-14748-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786501898713699-14748-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786501898713699-14748-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786501898713699-14748-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.103.62:37023
--webserver_interface=127.14.103.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.62.keytab
--principal=kudu/127.14.103.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.103.20:38123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.103.62:37023
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786501898713699-14748-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786501898713699-14748-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786501898713699-14748-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:36:45.873481 7353 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:36:45.873797 7353 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:36:45.873845 7353 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:36:45.877964 7353 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:36:45.878033 7353 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:36:45.878053 7353 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:36:45.878070 7353 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:36:45.878085 7353 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 02:36:45.883656 7353 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.103.20:38123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786501898713699-14748-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786501898713699-14748-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.103.62:37023
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.62.keytab
--principal=kudu/127.14.103.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.103.62:37023
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786501898713699-14748-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.103.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7353
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786501898713699-14748-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:36:45.884727 7353 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:36:45.885771 7353 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:36:45.894584 7363 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:36:45.894703 7360 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:36:45.894866 7353 server_base.cc:1034] running on GCE node
W20260812 02:36:45.895030 7359 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:36:45.895641 7353 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:36:45.896744 7353 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:36:45.897912 7353 hybrid_clock.cc:648] HybridClock initialized: now 1786502205897895 us; error 34 us; skew 500 ppm
Aug 12 02:36:45 dist-test-slave-2w3w krb5kdc[7339](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502205, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:36:45.900806 7353 init.cc:377] Logged in from keytab as kudu/127.14.103.62@KRBTEST.COM (short username kudu)
I20260812 02:36:45.901641 7353 webserver.cc:460] Webserver started at http://127.14.103.62:39281/ using document root <none> and password file <none>
I20260812 02:36:45.902156 7353 fs_manager.cc:359] Metadata directory not provided
I20260812 02:36:45.902215 7353 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:36:45.902381 7353 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:36:45.904438 7353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786501898713699-14748-0/minicluster-data/master-0/data/instance:
uuid: "274c9649a5bf48d1be1a557613f57f73"
format_stamp: "Formatted at 2026-08-12 02:36:45 on dist-test-slave-2w3w"
server_key: "71a77a32884b8c9ece06894c1be2b100"
server_key_iv: "12bbf0107ef346f49a02156c19c4d36a"
server_key_version: "encryptionkey@0"
I20260812 02:36:45.904958 7353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786501898713699-14748-0/minicluster-data/master-0/wal/instance:
uuid: "274c9649a5bf48d1be1a557613f57f73"
format_stamp: "Formatted at 2026-08-12 02:36:45 on dist-test-slave-2w3w"
server_key: "71a77a32884b8c9ece06894c1be2b100"
server_key_iv: "12bbf0107ef346f49a02156c19c4d36a"
server_key_version: "encryptionkey@0"
I20260812 02:36:45.908644 7353 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260812 02:36:45.911080 7371 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:36:45.912307 7353 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260812 02:36:45.912456 7353 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786501898713699-14748-0/minicluster-data/master-0/data,/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786501898713699-14748-0/minicluster-data/master-0/wal
uuid: "274c9649a5bf48d1be1a557613f57f73"
format_stamp: "Formatted at 2026-08-12 02:36:45 on dist-test-slave-2w3w"
server_key: "71a77a32884b8c9ece06894c1be2b100"
server_key_iv: "12bbf0107ef346f49a02156c19c4d36a"
server_key_version: "encryptionkey@0"
I20260812 02:36:45.912573 7353 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786501898713699-14748-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786501898713699-14748-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786501898713699-14748-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:36:45.934298 7353 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:36:45.937711 7353 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:36:45.937973 7353 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:36:45.946252 7353 rpc_server.cc:307] RPC server started. Bound to: 127.14.103.62:37023
I20260812 02:36:45.946342 7427 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.103.62:37023 every 8 connection(s)
I20260812 02:36:45.947505 7353 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786501898713699-14748-0/minicluster-data/master-0/data/info.pb
I20260812 02:36:45.950579 7429 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:36:45.953785 14748 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbep4M4/build/debug/bin/kudu as pid 7353
I20260812 02:36:45.953972 14748 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786501898713699-14748-0/minicluster-data/master-0/wal/instance
I20260812 02:36:45.954250 14748 external_mini_cluster.cc:1428] Setting key 5b8d5018a261a6b4e42ca36631c89b2a
I20260812 02:36:45.957087 7429 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 274c9649a5bf48d1be1a557613f57f73: Bootstrap starting.
I20260812 02:36:45.959884 7429 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 274c9649a5bf48d1be1a557613f57f73: Neither blocks nor log segments found. Creating new log.
I20260812 02:36:45.960916 7429 log.cc:824] T 00000000000000000000000000000000 P 274c9649a5bf48d1be1a557613f57f73: Log is configured to *not* fsync() on all Append() calls
I20260812 02:36:45.963347 7429 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 274c9649a5bf48d1be1a557613f57f73: No bootstrap required, opened a new log
Aug 12 02:36:45 dist-test-slave-2w3w krb5kdc[7339](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502205, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.103.62@KRBTEST.COM
I20260812 02:36:45.967044 7429 raft_consensus.cc:348] T 00000000000000000000000000000000 P 274c9649a5bf48d1be1a557613f57f73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "274c9649a5bf48d1be1a557613f57f73" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 37023 } }
I20260812 02:36:45.967360 7429 raft_consensus.cc:374] T 00000000000000000000000000000000 P 274c9649a5bf48d1be1a557613f57f73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:36:45.967451 7429 raft_consensus.cc:729] T 00000000000000000000000000000000 P 274c9649a5bf48d1be1a557613f57f73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 274c9649a5bf48d1be1a557613f57f73, State: Initialized, Role: FOLLOWER
I20260812 02:36:45.967981 7429 consensus_queue.cc:260] T 00000000000000000000000000000000 P 274c9649a5bf48d1be1a557613f57f73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "274c9649a5bf48d1be1a557613f57f73" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 37023 } }
I20260812 02:36:45.968152 7429 raft_consensus.cc:388] T 00000000000000000000000000000000 P 274c9649a5bf48d1be1a557613f57f73 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 02:36:45.968245 7429 raft_consensus.cc:482] T 00000000000000000000000000000000 P 274c9649a5bf48d1be1a557613f57f73 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 02:36:45.968371 7429 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 274c9649a5bf48d1be1a557613f57f73 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:36:45.969430 7429 raft_consensus.cc:504] T 00000000000000000000000000000000 P 274c9649a5bf48d1be1a557613f57f73 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "274c9649a5bf48d1be1a557613f57f73" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 37023 } }
I20260812 02:36:45.969816 7429 leader_election.cc:302] T 00000000000000000000000000000000 P 274c9649a5bf48d1be1a557613f57f73 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 274c9649a5bf48d1be1a557613f57f73; no voters:
I20260812 02:36:45.970221 7429 leader_election.cc:288] T 00000000000000000000000000000000 P 274c9649a5bf48d1be1a557613f57f73 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 02:36:45.970316 7435 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 274c9649a5bf48d1be1a557613f57f73 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:36:45.970567 7435 raft_consensus.cc:686] T 00000000000000000000000000000000 P 274c9649a5bf48d1be1a557613f57f73 [term 1 LEADER]: Becoming Leader. State: Replica: 274c9649a5bf48d1be1a557613f57f73, State: Running, Role: LEADER
I20260812 02:36:45.970996 7435 consensus_queue.cc:237] T 00000000000000000000000000000000 P 274c9649a5bf48d1be1a557613f57f73 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "274c9649a5bf48d1be1a557613f57f73" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 37023 } }
I20260812 02:36:45.971609 7433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:45.955745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33133 (local address 127.14.103.62:37023)
0812 02:36:45.956225 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0812 02:36:45.956244 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0812 02:36:45.956294 (+ 50us) server_negotiation.cc:407] Connection header received
0812 02:36:45.956956 (+ 662us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:36:45.956974 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:36:45.957395 (+ 421us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:36:45.957686 (+ 291us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:36:45.958702 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:45.959855 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:45.961197 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:45.961705 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:45.961914 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:36:45.964763 (+ 2849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:36:45.964769 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:36:45.964785 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:36:45.964826 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:36:45.967749 (+ 2923us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:36:45.968422 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:36:45.968428 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:36:45.968433 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:36:45.968616 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:36:45.968948 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:36:45.968951 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:36:45.968953 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:36:45.969445 (+ 492us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:36:45.969689 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0812 02:36:45.970017 (+ 328us) server_negotiation.cc:299] Negotiation successful
0812 02:36:45.970593 (+ 576us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":177,"threads_started":1}
I20260812 02:36:45.971678 7429 sys_catalog.cc:564] T 00000000000000000000000000000000 P 274c9649a5bf48d1be1a557613f57f73 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:36:45.972728 7437 sys_catalog.cc:455] T 00000000000000000000000000000000 P 274c9649a5bf48d1be1a557613f57f73 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 274c9649a5bf48d1be1a557613f57f73. Latest consensus state: current_term: 1 leader_uuid: "274c9649a5bf48d1be1a557613f57f73" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "274c9649a5bf48d1be1a557613f57f73" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 37023 } } }
I20260812 02:36:45.972893 7437 sys_catalog.cc:458] T 00000000000000000000000000000000 P 274c9649a5bf48d1be1a557613f57f73 [sys.catalog]: This master's current role is: LEADER
I20260812 02:36:45.973327 7445 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 02:36:45.973285 7438 sys_catalog.cc:455] T 00000000000000000000000000000000 P 274c9649a5bf48d1be1a557613f57f73 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "274c9649a5bf48d1be1a557613f57f73" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "274c9649a5bf48d1be1a557613f57f73" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 37023 } } }
I20260812 02:36:45.973462 7438 sys_catalog.cc:458] T 00000000000000000000000000000000 P 274c9649a5bf48d1be1a557613f57f73 [sys.catalog]: This master's current role is: LEADER
I20260812 02:36:45.975178 14748 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 02:36:45.976832 7445 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 02:36:45.983165 7445 catalog_manager.cc:1347] Generated new cluster ID: 384c4fbfd1334917b626f43f3030ad18
I20260812 02:36:45.983269 7445 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 02:36:45.987107 7433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:45.977814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33383 (local address 127.14.103.62:37023)
0812 02:36:45.977934 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0812 02:36:45.977939 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:36:45.978112 (+ 173us) server_negotiation.cc:407] Connection header received
0812 02:36:45.978334 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:36:45.978338 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:36:45.978384 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:36:45.978477 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:36:45.979271 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:45.980125 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:45.981172 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:45.981566 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:45.981695 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:36:45.983431 (+ 1736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:36:45.983436 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:36:45.983439 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:36:45.983470 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:36:45.985266 (+ 1796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:36:45.985761 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:36:45.985765 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:36:45.985768 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:36:45.985873 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:36:45.986157 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:36:45.986161 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:36:45.986163 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:36:45.986393 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:36:45.986531 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0812 02:36:45.986760 (+ 229us) server_negotiation.cc:299] Negotiation successful
0812 02:36:45.986910 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260812 02:36:45.991665 7445 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 02:36:45.992743 7445 catalog_manager.cc:1504] Loading token signing keys...
I20260812 02:36:45.996809 7445 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 274c9649a5bf48d1be1a557613f57f73: Generated new TSK 0
I20260812 02:36:45.997480 7445 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 02:36:46.002878 14748 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbep4M4/build/debug/bin/kudu with pid 7353
2026-08-12T02:36:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1543 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786501898713699-14748-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 02:36:46 dist-test-slave-2w3w krb5kdc[7464](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 02:36:46 dist-test-slave-2w3w krb5kdc[7464](info): set up 2 sockets
Aug 12 02:36:46 dist-test-slave-2w3w krb5kdc[7464](info): commencing operation
krb5kdc: starting...
W20260812 02:36:47.133711 14748 mini_kdc.cc:121] Time spent starting KDC: real 1.107s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786501898713699-14748-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786501898713699-14748-0/krb5kdc/test-user.keytab.
Aug 12 02:36:47 dist-test-slave-2w3w krb5kdc[7464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502207, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T02:36:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T02:36:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.103.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.103.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.103.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.62.keytab.
Entry for principal kudu/127.14.103.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.62.keytab.
I20260812 02:36:47.275393 14748 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbep4M4/build/debug/bin/kudu
/tmp/dist-test-taskbep4M4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786501898713699-14748-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786501898713699-14748-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786501898713699-14748-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786501898713699-14748-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.103.62:43599
--webserver_interface=127.14.103.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.62.keytab
--principal=kudu/127.14.103.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.103.20:35881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.103.62:43599
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786501898713699-14748-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786501898713699-14748-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786501898713699-14748-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:36:47.392207 7480 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:36:47.392560 7480 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:36:47.392660 7480 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:36:47.396790 7480 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:36:47.396904 7480 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:36:47.396943 7480 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:36:47.396982 7480 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:36:47.397018 7480 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 02:36:47.402527 7480 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.103.20:35881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786501898713699-14748-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786501898713699-14748-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.103.62:43599
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.62.keytab
--principal=kudu/127.14.103.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.103.62:43599
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786501898713699-14748-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.103.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7480
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786501898713699-14748-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:36:47.403715 7480 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:36:47.404798 7480 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:36:47.414055 7488 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:36:47.414109 7486 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:36:47.414111 7485 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:36:47.414228 7480 server_base.cc:1034] running on GCE node
I20260812 02:36:47.414951 7480 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:36:47.416153 7480 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:36:47.417342 7480 hybrid_clock.cc:648] HybridClock initialized: now 1786502207417317 us; error 49 us; skew 500 ppm
Aug 12 02:36:47 dist-test-slave-2w3w krb5kdc[7464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502207, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:36:47.420353 7480 init.cc:377] Logged in from keytab as kudu/127.14.103.62@KRBTEST.COM (short username kudu)
I20260812 02:36:47.421370 7480 webserver.cc:460] Webserver started at http://127.14.103.62:40045/ using document root <none> and password file <none>
I20260812 02:36:47.422006 7480 fs_manager.cc:359] Metadata directory not provided
I20260812 02:36:47.422104 7480 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:36:47.422369 7480 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:36:47.424734 7480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786501898713699-14748-0/minicluster-data/master-0/data/instance:
uuid: "2802bee9160d4ee08c9c6ea13e081fde"
format_stamp: "Formatted at 2026-08-12 02:36:47 on dist-test-slave-2w3w"
server_key: "c13754edc84f4103b1868f5873c436bb"
server_key_iv: "f4f053cd502866b424b89c86b09e1e32"
server_key_version: "encryptionkey@0"
I20260812 02:36:47.425376 7480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786501898713699-14748-0/minicluster-data/master-0/wal/instance:
uuid: "2802bee9160d4ee08c9c6ea13e081fde"
format_stamp: "Formatted at 2026-08-12 02:36:47 on dist-test-slave-2w3w"
server_key: "c13754edc84f4103b1868f5873c436bb"
server_key_iv: "f4f053cd502866b424b89c86b09e1e32"
server_key_version: "encryptionkey@0"
I20260812 02:36:47.429507 7480 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260812 02:36:47.432328 7495 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:36:47.433773 7480 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260812 02:36:47.433970 7480 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786501898713699-14748-0/minicluster-data/master-0/data,/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786501898713699-14748-0/minicluster-data/master-0/wal
uuid: "2802bee9160d4ee08c9c6ea13e081fde"
format_stamp: "Formatted at 2026-08-12 02:36:47 on dist-test-slave-2w3w"
server_key: "c13754edc84f4103b1868f5873c436bb"
server_key_iv: "f4f053cd502866b424b89c86b09e1e32"
server_key_version: "encryptionkey@0"
I20260812 02:36:47.434105 7480 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786501898713699-14748-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786501898713699-14748-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786501898713699-14748-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:36:47.445106 7480 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:36:47.448863 7480 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:36:47.449126 7480 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:36:47.456997 7480 rpc_server.cc:307] RPC server started. Bound to: 127.14.103.62:43599
I20260812 02:36:47.457010 7549 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.103.62:43599 every 8 connection(s)
I20260812 02:36:47.458086 7480 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786501898713699-14748-0/minicluster-data/master-0/data/info.pb
I20260812 02:36:47.461017 7550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:36:47.463636 14748 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbep4M4/build/debug/bin/kudu as pid 7480
I20260812 02:36:47.463804 14748 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786501898713699-14748-0/minicluster-data/master-0/wal/instance
I20260812 02:36:47.464105 14748 external_mini_cluster.cc:1428] Setting key eb1d7ec7e2656b299baca57259ee1c91
I20260812 02:36:47.467208 7550 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2802bee9160d4ee08c9c6ea13e081fde: Bootstrap starting.
I20260812 02:36:47.469408 7550 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2802bee9160d4ee08c9c6ea13e081fde: Neither blocks nor log segments found. Creating new log.
I20260812 02:36:47.470355 7550 log.cc:824] T 00000000000000000000000000000000 P 2802bee9160d4ee08c9c6ea13e081fde: Log is configured to *not* fsync() on all Append() calls
I20260812 02:36:47.472592 7550 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2802bee9160d4ee08c9c6ea13e081fde: No bootstrap required, opened a new log
I20260812 02:36:47.473340 7554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:47.465483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60939 (local address 127.14.103.62:43599)
0812 02:36:47.465923 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0812 02:36:47.465939 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0812 02:36:47.465974 (+ 35us) server_negotiation.cc:407] Connection header received
0812 02:36:47.466585 (+ 611us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:36:47.466601 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:36:47.466928 (+ 327us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:36:47.467320 (+ 392us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:36:47.467752 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:47.468717 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:47.469650 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:47.470371 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:47.470557 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:36:47.471417 (+ 860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:36:47.471423 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:36:47.471438 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:36:47.471917 (+ 479us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:36:47.471991 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:36:47.472023 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:36:47.472131 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0812 02:36:47.472227 (+ 96us) server_negotiation.cc:299] Negotiation successful
0812 02:36:47.472439 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":140,"threads_started":1}
I20260812 02:36:47.476379 14748 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 02:36:47.477083 7550 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2802bee9160d4ee08c9c6ea13e081fde [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2802bee9160d4ee08c9c6ea13e081fde" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 43599 } }
I20260812 02:36:47.477389 7550 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2802bee9160d4ee08c9c6ea13e081fde [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:36:47.477454 7550 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2802bee9160d4ee08c9c6ea13e081fde [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2802bee9160d4ee08c9c6ea13e081fde, State: Initialized, Role: FOLLOWER
I20260812 02:36:47.477952 7550 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2802bee9160d4ee08c9c6ea13e081fde [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2802bee9160d4ee08c9c6ea13e081fde" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 43599 } }
I20260812 02:36:47.478117 7550 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2802bee9160d4ee08c9c6ea13e081fde [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 02:36:47.478189 7550 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2802bee9160d4ee08c9c6ea13e081fde [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 02:36:47.478291 7550 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2802bee9160d4ee08c9c6ea13e081fde [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:36:47.479673 7550 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2802bee9160d4ee08c9c6ea13e081fde [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2802bee9160d4ee08c9c6ea13e081fde" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 43599 } }
I20260812 02:36:47.480042 7550 leader_election.cc:302] T 00000000000000000000000000000000 P 2802bee9160d4ee08c9c6ea13e081fde [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2802bee9160d4ee08c9c6ea13e081fde; no voters:
I20260812 02:36:47.480372 7550 leader_election.cc:288] T 00000000000000000000000000000000 P 2802bee9160d4ee08c9c6ea13e081fde [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 02:36:47.480444 7561 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2802bee9160d4ee08c9c6ea13e081fde [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:36:47.480667 7561 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2802bee9160d4ee08c9c6ea13e081fde [term 1 LEADER]: Becoming Leader. State: Replica: 2802bee9160d4ee08c9c6ea13e081fde, State: Running, Role: LEADER
I20260812 02:36:47.481119 7561 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2802bee9160d4ee08c9c6ea13e081fde [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2802bee9160d4ee08c9c6ea13e081fde" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 43599 } }
I20260812 02:36:47.481736 7550 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2802bee9160d4ee08c9c6ea13e081fde [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:36:47.482312 7561 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2802bee9160d4ee08c9c6ea13e081fde [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2802bee9160d4ee08c9c6ea13e081fde" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2802bee9160d4ee08c9c6ea13e081fde" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 43599 } } }
I20260812 02:36:47.482493 7561 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2802bee9160d4ee08c9c6ea13e081fde [sys.catalog]: This master's current role is: LEADER
I20260812 02:36:47.482985 7567 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 02:36:47.482897 7562 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2802bee9160d4ee08c9c6ea13e081fde [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2802bee9160d4ee08c9c6ea13e081fde. Latest consensus state: current_term: 1 leader_uuid: "2802bee9160d4ee08c9c6ea13e081fde" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2802bee9160d4ee08c9c6ea13e081fde" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 43599 } } }
I20260812 02:36:47.483299 7562 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2802bee9160d4ee08c9c6ea13e081fde [sys.catalog]: This master's current role is: LEADER
I20260812 02:36:47.483455 7554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:47.478687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39961 (local address 127.14.103.62:43599)
0812 02:36:47.478816 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0812 02:36:47.478820 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:36:47.479101 (+ 281us) server_negotiation.cc:407] Connection header received
0812 02:36:47.479233 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:36:47.479235 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:36:47.479274 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:36:47.479339 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:36:47.479674 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:47.480311 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:47.481415 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:47.481642 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:47.481754 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:36:47.482645 (+ 891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:36:47.482650 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:36:47.482654 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:36:47.482672 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:36:47.482710 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:36:47.482725 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:36:47.482801 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0812 02:36:47.483197 (+ 396us) server_negotiation.cc:299] Negotiation successful
0812 02:36:47.483260 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260812 02:36:47.487020 7567 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 02:36:47.493876 7567 catalog_manager.cc:1347] Generated new cluster ID: 28d143d226c044ea961f63b3d66dc513
I20260812 02:36:47.493990 7567 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 02:36:47.505571 7567 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 02:36:47.506635 7567 catalog_manager.cc:1504] Loading token signing keys...
I20260812 02:36:47.514453 7567 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2802bee9160d4ee08c9c6ea13e081fde: Generated new TSK 0
I20260812 02:36:47.515273 7567 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 02:36:47.519321 14748 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbep4M4/build/debug/bin/kudu with pid 7480
2026-08-12T02:36:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1515 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786501898713699-14748-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 02:36:47 dist-test-slave-2w3w krb5kdc[7586](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 02:36:47 dist-test-slave-2w3w krb5kdc[7586](info): set up 2 sockets
Aug 12 02:36:47 dist-test-slave-2w3w krb5kdc[7586](info): commencing operation
krb5kdc: starting...
W20260812 02:36:48.629113 14748 mini_kdc.cc:121] Time spent starting KDC: real 1.087s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786501898713699-14748-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786501898713699-14748-0/krb5kdc/test-user.keytab.
Aug 12 02:36:48 dist-test-slave-2w3w krb5kdc[7586](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502208, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T02:36:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T02:36:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.103.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.103.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.103.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.62.keytab.
Entry for principal kudu/127.14.103.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.62.keytab.
I20260812 02:36:48.770588 14748 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbep4M4/build/debug/bin/kudu
/tmp/dist-test-taskbep4M4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786501898713699-14748-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786501898713699-14748-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786501898713699-14748-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786501898713699-14748-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.103.62:39671
--webserver_interface=127.14.103.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.62.keytab
--principal=kudu/127.14.103.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.103.20:32833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.103.62:39671
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786501898713699-14748-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786501898713699-14748-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786501898713699-14748-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:36:48.886075 7599 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:36:48.886407 7599 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:36:48.886453 7599 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:36:48.890755 7599 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:36:48.890841 7599 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:36:48.890862 7599 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:36:48.890879 7599 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:36:48.890895 7599 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 02:36:48.896553 7599 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.103.20:32833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786501898713699-14748-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786501898713699-14748-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.103.62:39671
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.62.keytab
--principal=kudu/127.14.103.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.103.62:39671
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786501898713699-14748-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.103.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7599
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786501898713699-14748-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:36:48.897799 7599 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:36:48.898856 7599 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:36:48.907841 7607 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:36:48.907846 7605 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:36:48.907846 7604 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:36:48.908138 7599 server_base.cc:1034] running on GCE node
I20260812 02:36:48.908784 7599 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:36:48.909960 7599 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:36:48.911157 7599 hybrid_clock.cc:648] HybridClock initialized: now 1786502208911144 us; error 27 us; skew 500 ppm
Aug 12 02:36:48 dist-test-slave-2w3w krb5kdc[7586](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502208, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:36:48.914111 7599 init.cc:377] Logged in from keytab as kudu/127.14.103.62@KRBTEST.COM (short username kudu)
I20260812 02:36:48.915203 7599 webserver.cc:460] Webserver started at http://127.14.103.62:46113/ using document root <none> and password file <none>
I20260812 02:36:48.915782 7599 fs_manager.cc:359] Metadata directory not provided
I20260812 02:36:48.915886 7599 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:36:48.916159 7599 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:36:48.918401 7599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786501898713699-14748-0/minicluster-data/master-0/data/instance:
uuid: "8217100a5a224ab385f74776d353ccba"
format_stamp: "Formatted at 2026-08-12 02:36:48 on dist-test-slave-2w3w"
server_key: "cbb541e3d31302d51b69e19cff9a15f2"
server_key_iv: "b6ecc99defca690db0ed8cfa33a3e8e5"
server_key_version: "encryptionkey@0"
I20260812 02:36:48.919063 7599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786501898713699-14748-0/minicluster-data/master-0/wal/instance:
uuid: "8217100a5a224ab385f74776d353ccba"
format_stamp: "Formatted at 2026-08-12 02:36:48 on dist-test-slave-2w3w"
server_key: "cbb541e3d31302d51b69e19cff9a15f2"
server_key_iv: "b6ecc99defca690db0ed8cfa33a3e8e5"
server_key_version: "encryptionkey@0"
I20260812 02:36:48.922868 7599 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.000s
I20260812 02:36:48.925388 7614 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:36:48.926678 7599 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260812 02:36:48.926843 7599 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786501898713699-14748-0/minicluster-data/master-0/data,/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786501898713699-14748-0/minicluster-data/master-0/wal
uuid: "8217100a5a224ab385f74776d353ccba"
format_stamp: "Formatted at 2026-08-12 02:36:48 on dist-test-slave-2w3w"
server_key: "cbb541e3d31302d51b69e19cff9a15f2"
server_key_iv: "b6ecc99defca690db0ed8cfa33a3e8e5"
server_key_version: "encryptionkey@0"
I20260812 02:36:48.926990 7599 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786501898713699-14748-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786501898713699-14748-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786501898713699-14748-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:36:48.932083 7599 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:36:48.935410 7599 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:36:48.935626 7599 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:36:48.943596 7599 rpc_server.cc:307] RPC server started. Bound to: 127.14.103.62:39671
I20260812 02:36:48.943626 7672 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.103.62:39671 every 8 connection(s)
I20260812 02:36:48.944599 7599 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786501898713699-14748-0/minicluster-data/master-0/data/info.pb
I20260812 02:36:48.947362 7673 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:36:48.948671 14748 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbep4M4/build/debug/bin/kudu as pid 7599
I20260812 02:36:48.948832 14748 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786501898713699-14748-0/minicluster-data/master-0/wal/instance
I20260812 02:36:48.949254 14748 external_mini_cluster.cc:1428] Setting key e19f6bc9f93928ff3143cbb6d5b03fd8
I20260812 02:36:48.953976 7673 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8217100a5a224ab385f74776d353ccba: Bootstrap starting.
I20260812 02:36:48.954795 7676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:48.950787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34039 (local address 127.14.103.62:39671)
0812 02:36:48.951364 (+ 577us) server_negotiation.cc:206] Beginning negotiation
0812 02:36:48.951379 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0812 02:36:48.951419 (+ 40us) server_negotiation.cc:407] Connection header received
0812 02:36:48.952055 (+ 636us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:36:48.952072 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:36:48.952357 (+ 285us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:36:48.952696 (+ 339us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:36:48.953533 (+ 837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:36:48.953538 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:36:48.953549 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:36:48.953561 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:36:48.953615 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:36:48.953633 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:36:48.953781 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0812 02:36:48.953851 (+ 70us) server_negotiation.cc:299] Negotiation successful
0812 02:36:48.954048 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":419,"thread_start_us":158,"threads_started":1}
I20260812 02:36:48.956548 7673 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8217100a5a224ab385f74776d353ccba: Neither blocks nor log segments found. Creating new log.
I20260812 02:36:48.957473 7673 log.cc:824] T 00000000000000000000000000000000 P 8217100a5a224ab385f74776d353ccba: Log is configured to *not* fsync() on all Append() calls
I20260812 02:36:48.957782 14748 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 02:36:48.959540 7673 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8217100a5a224ab385f74776d353ccba: No bootstrap required, opened a new log
I20260812 02:36:48.961587 7676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:48.959946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55865 (local address 127.14.103.62:39671)
0812 02:36:48.960074 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0812 02:36:48.960079 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:36:48.960301 (+ 222us) server_negotiation.cc:407] Connection header received
0812 02:36:48.960384 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:36:48.960386 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:36:48.960417 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:36:48.960488 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:36:48.961204 (+ 716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:36:48.961207 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:36:48.961209 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 02:36:48.961219 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:36:48.961243 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 02:36:48.961253 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:36:48.961324 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0812 02:36:48.961416 (+ 92us) server_negotiation.cc:299] Negotiation successful
0812 02:36:48.961447 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260812 02:36:48.964067 7673 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8217100a5a224ab385f74776d353ccba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8217100a5a224ab385f74776d353ccba" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 39671 } }
I20260812 02:36:48.964363 7673 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8217100a5a224ab385f74776d353ccba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:36:48.964430 7673 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8217100a5a224ab385f74776d353ccba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8217100a5a224ab385f74776d353ccba, State: Initialized, Role: FOLLOWER
I20260812 02:36:48.964848 7673 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8217100a5a224ab385f74776d353ccba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8217100a5a224ab385f74776d353ccba" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 39671 } }
I20260812 02:36:48.965004 7673 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8217100a5a224ab385f74776d353ccba [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 02:36:48.965077 7673 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8217100a5a224ab385f74776d353ccba [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 02:36:48.965200 7673 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8217100a5a224ab385f74776d353ccba [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:36:48.966163 7673 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8217100a5a224ab385f74776d353ccba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8217100a5a224ab385f74776d353ccba" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 39671 } }
I20260812 02:36:48.966513 7673 leader_election.cc:302] T 00000000000000000000000000000000 P 8217100a5a224ab385f74776d353ccba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8217100a5a224ab385f74776d353ccba; no voters:
I20260812 02:36:48.966861 7673 leader_election.cc:288] T 00000000000000000000000000000000 P 8217100a5a224ab385f74776d353ccba [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 02:36:48.967021 7683 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8217100a5a224ab385f74776d353ccba [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:36:48.967298 7683 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8217100a5a224ab385f74776d353ccba [term 1 LEADER]: Becoming Leader. State: Replica: 8217100a5a224ab385f74776d353ccba, State: Running, Role: LEADER
I20260812 02:36:48.967679 7683 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8217100a5a224ab385f74776d353ccba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8217100a5a224ab385f74776d353ccba" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 39671 } }
I20260812 02:36:48.968212 7673 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8217100a5a224ab385f74776d353ccba [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:36:48.969282 7684 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8217100a5a224ab385f74776d353ccba [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8217100a5a224ab385f74776d353ccba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8217100a5a224ab385f74776d353ccba" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 39671 } } }
I20260812 02:36:48.969704 7684 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8217100a5a224ab385f74776d353ccba [sys.catalog]: This master's current role is: LEADER
I20260812 02:36:48.969594 7685 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8217100a5a224ab385f74776d353ccba [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8217100a5a224ab385f74776d353ccba. Latest consensus state: current_term: 1 leader_uuid: "8217100a5a224ab385f74776d353ccba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8217100a5a224ab385f74776d353ccba" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 39671 } } }
I20260812 02:36:48.970059 7685 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8217100a5a224ab385f74776d353ccba [sys.catalog]: This master's current role is: LEADER
I20260812 02:36:48.970168 7694 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 02:36:48.973560 7694 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 02:36:48.979589 7694 catalog_manager.cc:1347] Generated new cluster ID: 6d1988e2a06641408bed626a48bc622c
I20260812 02:36:48.979688 7694 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 02:36:48.992166 7694 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 02:36:48.992583 7694 catalog_manager.cc:1504] Loading token signing keys...
I20260812 02:36:48.997040 7694 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8217100a5a224ab385f74776d353ccba: Generated new TSK 0
I20260812 02:36:48.998029 7694 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 02:36:49.012226 14748 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbep4M4/build/debug/bin/kudu with pid 7599
2026-08-12T02:36:49Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1493 ms)
[----------] 18 tests from AuthTokenIssuingTest (27710 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786501898713699-14748-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 02:36:49 dist-test-slave-2w3w krb5kdc[7709](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 02:36:49 dist-test-slave-2w3w krb5kdc[7709](info): set up 2 sockets
Aug 12 02:36:49 dist-test-slave-2w3w krb5kdc[7709](info): commencing operation
krb5kdc: starting...
W20260812 02:36:50.150770 14748 mini_kdc.cc:121] Time spent starting KDC: real 1.114s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786501898713699-14748-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786501898713699-14748-0/krb5kdc/test-user.keytab.
Aug 12 02:36:50 dist-test-slave-2w3w krb5kdc[7709](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502210, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T02:36:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T02:36:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.103.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.103.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.103.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.62.keytab.
Entry for principal kudu/127.14.103.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.62.keytab.
I20260812 02:36:50.311542 14748 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbep4M4/build/debug/bin/kudu
/tmp/dist-test-taskbep4M4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786501898713699-14748-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786501898713699-14748-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786501898713699-14748-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786501898713699-14748-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.103.62:39137
--webserver_interface=127.14.103.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.62.keytab
--principal=kudu/127.14.103.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.103.20:36591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.103.62:39137,127.14.103.61:35237,127.14.103.60:45435
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786501898713699-14748-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786501898713699-14748-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786501898713699-14748-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:36:50.425168 7722 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260812 02:36:50.425504 7722 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:36:50.425530 7722 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:36:50.425623 7722 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:36:50.429926 7722 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:36:50.430006 7722 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:36:50.430027 7722 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:36:50.430045 7722 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:36:50.430061 7722 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 02:36:50.436013 7722 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.103.20:36591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786501898713699-14748-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786501898713699-14748-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.103.62:39137,127.14.103.61:35237,127.14.103.60:45435
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.62.keytab
--principal=kudu/127.14.103.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.103.62:39137
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786501898713699-14748-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.103.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7722
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786501898713699-14748-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:36:50.437342 7722 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:36:50.438421 7722 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:36:50.447481 7732 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:36:50.447479 7730 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:36:50.447479 7728 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:36:50.448001 7722 server_base.cc:1034] running on GCE node
I20260812 02:36:50.448658 7722 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:36:50.449923 7722 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:36:50.451071 7722 hybrid_clock.cc:648] HybridClock initialized: now 1786502210451052 us; error 37 us; skew 500 ppm
Aug 12 02:36:50 dist-test-slave-2w3w krb5kdc[7709](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502210, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:36:50.454362 7722 init.cc:377] Logged in from keytab as kudu/127.14.103.62@KRBTEST.COM (short username kudu)
I20260812 02:36:50.455521 7722 webserver.cc:460] Webserver started at http://127.14.103.62:39291/ using document root <none> and password file <none>
I20260812 02:36:50.456127 7722 fs_manager.cc:359] Metadata directory not provided
I20260812 02:36:50.456223 7722 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:36:50.456473 7722 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:36:50.458858 7722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786501898713699-14748-0/minicluster-data/master-0/data/instance:
uuid: "6d4628539e294e919d4e53bea3a836c9"
format_stamp: "Formatted at 2026-08-12 02:36:50 on dist-test-slave-2w3w"
server_key: "2c740ed91242d65816ad0799844ed2e8"
server_key_iv: "d01fff7c949a2665f46709f6e3ecfda4"
server_key_version: "encryptionkey@0"
I20260812 02:36:50.459573 7722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786501898713699-14748-0/minicluster-data/master-0/wal/instance:
uuid: "6d4628539e294e919d4e53bea3a836c9"
format_stamp: "Formatted at 2026-08-12 02:36:50 on dist-test-slave-2w3w"
server_key: "2c740ed91242d65816ad0799844ed2e8"
server_key_iv: "d01fff7c949a2665f46709f6e3ecfda4"
server_key_version: "encryptionkey@0"
I20260812 02:36:50.463831 7722 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260812 02:36:50.466602 7739 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:36:50.468091 7722 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 02:36:50.468269 7722 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786501898713699-14748-0/minicluster-data/master-0/data,/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786501898713699-14748-0/minicluster-data/master-0/wal
uuid: "6d4628539e294e919d4e53bea3a836c9"
format_stamp: "Formatted at 2026-08-12 02:36:50 on dist-test-slave-2w3w"
server_key: "2c740ed91242d65816ad0799844ed2e8"
server_key_iv: "d01fff7c949a2665f46709f6e3ecfda4"
server_key_version: "encryptionkey@0"
I20260812 02:36:50.468410 7722 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786501898713699-14748-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786501898713699-14748-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786501898713699-14748-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:36:50.481663 7722 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:36:50.485208 7722 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:36:50.485440 7722 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:36:50.493111 7722 rpc_server.cc:307] RPC server started. Bound to: 127.14.103.62:39137
I20260812 02:36:50.493395 7797 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.103.62:39137 every 8 connection(s)
I20260812 02:36:50.494167 7722 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786501898713699-14748-0/minicluster-data/master-0/data/info.pb
I20260812 02:36:50.497228 7799 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:36:50.499351 14748 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbep4M4/build/debug/bin/kudu as pid 7722
I20260812 02:36:50.499511 14748 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786501898713699-14748-0/minicluster-data/master-0/wal/instance
I20260812 02:36:50.499786 14748 external_mini_cluster.cc:1428] Setting key 065e24f33868fc723c872db3ae64f8c2
I20260812 02:36:50.501243 7799 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.103.62" port: 39137 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.14.103.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.103.61@KRBTEST.COM" created.
Aug 12 02:36:50 dist-test-slave-2w3w krb5kdc[7709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502210, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.62@KRBTEST.COM for kudu/127.14.103.62@KRBTEST.COM
I20260812 02:36:50.519006 7802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:50.504078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39908 (local address 127.14.103.62:39137)
0812 02:36:50.504645 (+ 567us) server_negotiation.cc:206] Beginning negotiation
0812 02:36:50.504667 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0812 02:36:50.504944 (+ 277us) server_negotiation.cc:407] Connection header received
0812 02:36:50.506050 (+ 1106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:36:50.506059 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:36:50.506328 (+ 269us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:36:50.506550 (+ 222us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:36:50.507460 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:50.508933 (+ 1473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:50.510035 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:50.510440 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:50.510577 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:36:50.513959 (+ 3382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:36:50.513965 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:36:50.513976 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:36:50.513997 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:36:50.515713 (+ 1716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:36:50.516224 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:36:50.516229 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:36:50.516231 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:36:50.516341 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:36:50.516660 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:36:50.516665 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:36:50.516667 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:36:50.517057 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:36:50.517198 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0812 02:36:50.517562 (+ 364us) server_negotiation.cc:299] Negotiation successful
0812 02:36:50.517896 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":25,"server-negotiator.queue_time_us":377,"thread_start_us":162,"threads_started":1}
I20260812 02:36:50.519006 7803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:50.504078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:39137 (local address 127.0.0.1:39908)
0812 02:36:50.504756 (+ 678us) negotiation.cc:107] Waiting for socket to connect
0812 02:36:50.504806 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0812 02:36:50.505689 (+ 883us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:36:50.506561 (+ 872us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:36:50.506565 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:36:50.506920 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:36:50.507273 (+ 353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:36:50.507286 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:36:50.509093 (+ 1807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:36:50.509096 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:36:50.509908 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:36:50.509917 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:36:50.510639 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:36:50.510644 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:36:50.510800 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:36:50.511458 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:36:50.511475 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:36:50.513774 (+ 2299us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:36:50.515847 (+ 2073us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:36:50.515866 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:36:50.515877 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:36:50.516107 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:36:50.516451 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:36:50.516454 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:36:50.516456 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:36:50.516568 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:36:50.517179 (+ 611us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:36:50.517185 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:36:50.517323 (+ 138us) client_negotiation.cc:769] Sending connection context
0812 02:36:50.517500 (+ 177us) client_negotiation.cc:240] Negotiation successful
0812 02:36:50.517948 (+ 448us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"mutex_wait_us":160,"spinlock_wait_cycles":25344,"thread_start_us":182,"threads_started":1}
I20260812 02:36:50.521693 7799 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.103.61" port: 35237 } has no permanent_uuid. Determining permanent_uuid...
I20260812 02:36:50.522493 7803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:50.522194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.61:35237 (local address 127.0.0.1:40248)
0812 02:36:50.522314 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0812 02:36:50.522391 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.61:35237: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260812 02:36:50.522708 7742 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.103.61:35237: connect: Connection refused (error 111)
I20260812 02:36:50.523797 7803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:50.523577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.61:35237 (local address 127.0.0.1:40254)
0812 02:36:50.523691 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0812 02:36:50.523734 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.61:35237: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260812 02:36:50.524240 7799 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.103.61:35237: Network error: Client connection negotiation failed: client connection to 127.14.103.61:35237: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.103.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.61.keytab.
Entry for principal kudu/127.14.103.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.61.keytab.
I20260812 02:36:50.530663 14748 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbep4M4/build/debug/bin/kudu
/tmp/dist-test-taskbep4M4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786501898713699-14748-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786501898713699-14748-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786501898713699-14748-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786501898713699-14748-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.103.61:35237
--webserver_interface=127.14.103.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.61.keytab
--principal=kudu/127.14.103.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.103.20:36591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.103.62:39137,127.14.103.61:35237,127.14.103.60:45435
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786501898713699-14748-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786501898713699-14748-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786501898713699-14748-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 02:36:50.573570 7799 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.103.61" port: 35237 } attempt: 1
I20260812 02:36:50.574580 7803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:50.574246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.61:35237 (local address 127.0.0.1:40262)
0812 02:36:50.574437 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0812 02:36:50.574502 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.61:35237: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260812 02:36:50.575469 7803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:50.575240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.61:35237 (local address 127.0.0.1:40272)
0812 02:36:50.575343 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0812 02:36:50.575408 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.61:35237: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260812 02:36:50.575650 7799 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.103.61:35237: Network error: Client connection negotiation failed: client connection to 127.14.103.61:35237: connect: Connection refused (error 111)
I20260812 02:36:50.643839 7799 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.103.61" port: 35237 } attempt: 2
I20260812 02:36:50.644693 7803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:50.644399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.61:35237 (local address 127.0.0.1:40288)
0812 02:36:50.644552 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0812 02:36:50.644619 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.61:35237: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260812 02:36:50.645493 7803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:50.645255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.61:35237 (local address 127.0.0.1:40304)
0812 02:36:50.645391 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0812 02:36:50.645436 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.61:35237: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260812 02:36:50.645624 7799 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.103.61:35237: Network error: Client connection negotiation failed: client connection to 127.14.103.61:35237: connect: Connection refused (error 111)
W20260812 02:36:50.645615 7807 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260812 02:36:50.645982 7807 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:36:50.646051 7807 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:36:50.646123 7807 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:36:50.650197 7807 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:36:50.650292 7807 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:36:50.650339 7807 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:36:50.650372 7807 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:36:50.650405 7807 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 02:36:50.656862 7807 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.103.20:36591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786501898713699-14748-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786501898713699-14748-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.14.103.62:39137,127.14.103.61:35237,127.14.103.60:45435
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.61.keytab
--principal=kudu/127.14.103.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.103.61:35237
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786501898713699-14748-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.14.103.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7807
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786501898713699-14748-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:36:50.657975 7807 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:36:50.659060 7807 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:36:50.667610 7813 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:36:50.667665 7807 server_base.cc:1034] running on GCE node
W20260812 02:36:50.667735 7812 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:36:50.667625 7815 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:36:50.668452 7807 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:36:50.669633 7807 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:36:50.670871 7807 hybrid_clock.cc:648] HybridClock initialized: now 1786502210670862 us; error 40 us; skew 500 ppm
Aug 12 02:36:50 dist-test-slave-2w3w krb5kdc[7709](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502210, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:36:50.674108 7807 init.cc:377] Logged in from keytab as kudu/127.14.103.61@KRBTEST.COM (short username kudu)
I20260812 02:36:50.675208 7807 webserver.cc:460] Webserver started at http://127.14.103.61:42251/ using document root <none> and password file <none>
I20260812 02:36:50.675800 7807 fs_manager.cc:359] Metadata directory not provided
I20260812 02:36:50.675894 7807 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:36:50.676132 7807 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:36:50.678238 7807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786501898713699-14748-0/minicluster-data/master-1/data/instance:
uuid: "d52e652d44204f9aad62c52c10a3f0fd"
format_stamp: "Formatted at 2026-08-12 02:36:50 on dist-test-slave-2w3w"
server_key: "e277f189d7fe4158a4da211f7e0e7ceb"
server_key_iv: "6a47a5b652c19622dbdbe81a4887bab8"
server_key_version: "encryptionkey@0"
I20260812 02:36:50.678853 7807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786501898713699-14748-0/minicluster-data/master-1/wal/instance:
uuid: "d52e652d44204f9aad62c52c10a3f0fd"
format_stamp: "Formatted at 2026-08-12 02:36:50 on dist-test-slave-2w3w"
server_key: "e277f189d7fe4158a4da211f7e0e7ceb"
server_key_iv: "6a47a5b652c19622dbdbe81a4887bab8"
server_key_version: "encryptionkey@0"
I20260812 02:36:50.682814 7807 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260812 02:36:50.685575 7822 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:36:50.687181 7807 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 02:36:50.687358 7807 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786501898713699-14748-0/minicluster-data/master-1/data,/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786501898713699-14748-0/minicluster-data/master-1/wal
uuid: "d52e652d44204f9aad62c52c10a3f0fd"
format_stamp: "Formatted at 2026-08-12 02:36:50 on dist-test-slave-2w3w"
server_key: "e277f189d7fe4158a4da211f7e0e7ceb"
server_key_iv: "6a47a5b652c19622dbdbe81a4887bab8"
server_key_version: "encryptionkey@0"
I20260812 02:36:50.687496 7807 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786501898713699-14748-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786501898713699-14748-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786501898713699-14748-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:36:50.704139 7807 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:36:50.707567 7807 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:36:50.707799 7807 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:36:50.715183 7807 rpc_server.cc:307] RPC server started. Bound to: 127.14.103.61:35237
I20260812 02:36:50.715508 7883 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.103.61:35237 every 8 connection(s)
I20260812 02:36:50.716209 7807 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786501898713699-14748-0/minicluster-data/master-1/data/info.pb
I20260812 02:36:50.718474 14748 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbep4M4/build/debug/bin/kudu as pid 7807
I20260812 02:36:50.718617 14748 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786501898713699-14748-0/minicluster-data/master-1/wal/instance
I20260812 02:36:50.718982 14748 external_mini_cluster.cc:1428] Setting key c85ddba3fdd46b728ef00b35542456c1
I20260812 02:36:50.719317 7884 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:36:50.723541 7884 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.103.62" port: 39137 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.14.103.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.103.60@KRBTEST.COM" created.
Aug 12 02:36:50 dist-test-slave-2w3w krb5kdc[7709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502210, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.61@KRBTEST.COM for kudu/127.14.103.62@KRBTEST.COM
I20260812 02:36:50.736788 7799 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.103.61" port: 35237 } attempt: 3
I20260812 02:36:50.740089 7802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:50.725567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39916 (local address 127.14.103.62:39137)
0812 02:36:50.725711 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0812 02:36:50.725714 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 02:36:50.726763 (+ 1049us) server_negotiation.cc:407] Connection header received
0812 02:36:50.727750 (+ 987us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:36:50.727753 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:36:50.727814 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:36:50.727928 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:36:50.729501 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:50.730090 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:50.731690 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:50.731930 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:50.732023 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:36:50.734776 (+ 2753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:36:50.734783 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:36:50.734789 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:36:50.734826 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:36:50.736655 (+ 1829us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:36:50.737643 (+ 988us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:36:50.737647 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:36:50.737649 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:36:50.737747 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:36:50.738148 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:36:50.738152 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:36:50.738153 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:36:50.738378 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:36:50.738479 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0812 02:36:50.739801 (+ 1322us) server_negotiation.cc:299] Negotiation successful
0812 02:36:50.739946 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260812 02:36:50.740598 7887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:50.726060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:39137 (local address 127.0.0.1:39916)
0812 02:36:50.726595 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0812 02:36:50.726643 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0812 02:36:50.727544 (+ 901us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:36:50.728108 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:36:50.728115 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:36:50.728525 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:36:50.729287 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:36:50.729307 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:36:50.730227 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:36:50.730229 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:36:50.731550 (+ 1321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:36:50.731565 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:36:50.732108 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:36:50.732114 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:36:50.732351 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:36:50.733042 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:36:50.733071 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:36:50.734576 (+ 1505us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:36:50.737088 (+ 2512us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:36:50.737093 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:36:50.737106 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:36:50.737506 (+ 400us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:36:50.737890 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:36:50.737892 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:36:50.737893 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:36:50.738032 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:36:50.738541 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:36:50.738549 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:36:50.738887 (+ 338us) client_negotiation.cc:769] Sending connection context
0812 02:36:50.739129 (+ 242us) client_negotiation.cc:240] Negotiation successful
0812 02:36:50.739530 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":161,"threads_started":1}
I20260812 02:36:50.742184 7884 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.103.61" port: 35237 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 02:36:50 dist-test-slave-2w3w krb5kdc[7709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502210, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.62@KRBTEST.COM for kudu/127.14.103.61@KRBTEST.COM
Aug 12 02:36:50 dist-test-slave-2w3w krb5kdc[7709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502210, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.61@KRBTEST.COM for kudu/127.14.103.61@KRBTEST.COM
I20260812 02:36:50.748512 7803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:50.737286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.61:35237 (local address 127.0.0.1:40320)
0812 02:36:50.737511 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0812 02:36:50.737530 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 02:36:50.737618 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:36:50.738431 (+ 813us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:36:50.738436 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:36:50.738611 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:36:50.738727 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:36:50.738733 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:36:50.740555 (+ 1822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:36:50.740559 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:36:50.741299 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:36:50.741308 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:36:50.742142 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:36:50.742146 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:36:50.742262 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:36:50.742811 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:36:50.742827 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:36:50.744486 (+ 1659us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:36:50.746664 (+ 2178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:36:50.746668 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:36:50.746671 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:36:50.746938 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:36:50.747277 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:36:50.747279 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:36:50.747280 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:36:50.747432 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:36:50.747908 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:36:50.747911 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:36:50.748023 (+ 112us) client_negotiation.cc:769] Sending connection context
0812 02:36:50.748143 (+ 120us) client_negotiation.cc:240] Negotiation successful
0812 02:36:50.748300 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128}
I20260812 02:36:50.749349 7889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:50.737383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40320 (local address 127.14.103.61:35237)
0812 02:36:50.737914 (+ 531us) server_negotiation.cc:206] Beginning negotiation
0812 02:36:50.737944 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0812 02:36:50.737974 (+ 30us) server_negotiation.cc:407] Connection header received
0812 02:36:50.738029 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:36:50.738038 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:36:50.738279 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:36:50.738429 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:36:50.738859 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:50.740423 (+ 1564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:50.741422 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:50.741961 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:50.742095 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:36:50.744613 (+ 2518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:36:50.744623 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:36:50.744636 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:36:50.744660 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:36:50.746552 (+ 1892us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:36:50.747089 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:36:50.747096 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:36:50.747098 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:36:50.747182 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:36:50.747538 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:36:50.747541 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:36:50.747542 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:36:50.747788 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:36:50.747967 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0812 02:36:50.748255 (+ 288us) server_negotiation.cc:299] Negotiation successful
0812 02:36:50.748399 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":400,"thread_start_us":154,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.103.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.60.keytab.
Entry for principal kudu/127.14.103.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.60.keytab.
I20260812 02:36:50.751606 7799 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.103.60" port: 45435 } has no permanent_uuid. Determining permanent_uuid...
I20260812 02:36:50.752235 14748 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbep4M4/build/debug/bin/kudu
/tmp/dist-test-taskbep4M4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786501898713699-14748-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786501898713699-14748-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786501898713699-14748-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786501898713699-14748-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.103.60:45435
--webserver_interface=127.14.103.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.60.keytab
--principal=kudu/127.14.103.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.103.20:36591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.103.62:39137,127.14.103.61:35237,127.14.103.60:45435
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786501898713699-14748-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786501898713699-14748-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786501898713699-14748-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 02:36:50.752430 7803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:50.752118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.60:45435 (local address 127.0.0.1:43764)
0812 02:36:50.752246 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0812 02:36:50.752308 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.60:45435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260812 02:36:50.752450 7887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:50.742805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.61:35237 (local address 127.0.0.1:40336)
0812 02:36:50.742923 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0812 02:36:50.742936 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 02:36:50.743074 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:36:50.743627 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:36:50.743632 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:36:50.743799 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:36:50.743941 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:36:50.743948 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:36:50.744842 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:36:50.744845 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:36:50.745440 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:36:50.745450 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:36:50.746203 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:36:50.746209 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:36:50.746339 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:36:50.746788 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:36:50.746807 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:36:50.748278 (+ 1471us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:36:50.750544 (+ 2266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:36:50.750548 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:36:50.750550 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:36:50.750800 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:36:50.751319 (+ 519us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:36:50.751340 (+ 21us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:36:50.751345 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:36:50.751456 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:36:50.751913 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:36:50.751917 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:36:50.752011 (+ 94us) client_negotiation.cc:769] Sending connection context
0812 02:36:50.752100 (+ 89us) client_negotiation.cc:240] Negotiation successful
0812 02:36:50.752246 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260812 02:36:50.753192 7890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:50.742952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40336 (local address 127.14.103.61:35237)
0812 02:36:50.743341 (+ 389us) server_negotiation.cc:206] Beginning negotiation
0812 02:36:50.743351 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 02:36:50.743368 (+ 17us) server_negotiation.cc:407] Connection header received
0812 02:36:50.743405 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:36:50.743410 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:36:50.743482 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:36:50.743584 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:36:50.744078 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:50.744743 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:50.745556 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:50.745925 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:50.746074 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:36:50.748436 (+ 2362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:36:50.748439 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:36:50.748441 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:36:50.748460 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:36:50.750383 (+ 1923us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:36:50.750950 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:36:50.750972 (+ 22us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:36:50.750977 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:36:50.751065 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:36:50.751582 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:36:50.751584 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:36:50.751587 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:36:50.751802 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:36:50.752727 (+ 925us) server_negotiation.cc:1035] Waiting for connection context
0812 02:36:50.752807 (+ 80us) server_negotiation.cc:299] Negotiation successful
0812 02:36:50.752939 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":144,"threads_started":1}
I20260812 02:36:50.753283 7803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:50.753012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.60:45435 (local address 127.0.0.1:43772)
0812 02:36:50.753152 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0812 02:36:50.753197 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.60:45435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
W20260812 02:36:50.753566 7799 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.103.60:45435: Network error: Client connection negotiation failed: client connection to 127.14.103.60:45435: connect: Connection refused (error 111)
I20260812 02:36:50.753852 7884 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.103.60" port: 45435 } has no permanent_uuid. Determining permanent_uuid...
I20260812 02:36:50.754528 7887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:50.754286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.60:45435 (local address 127.0.0.1:43774)
0812 02:36:50.754413 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0812 02:36:50.754480 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.60:45435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260812 02:36:50.754846 7827 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.103.60:45435: connect: Connection refused (error 111)
I20260812 02:36:50.755725 7887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:50.755518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.60:45435 (local address 127.0.0.1:43786)
0812 02:36:50.755628 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0812 02:36:50.755671 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.60:45435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260812 02:36:50.755861 7884 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.103.60:45435: Network error: Client connection negotiation failed: client connection to 127.14.103.60:45435: connect: Connection refused (error 111)
I20260812 02:36:50.784801 7799 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.103.60" port: 45435 } attempt: 1
I20260812 02:36:50.785727 7803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:50.785409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.60:45435 (local address 127.0.0.1:43800)
0812 02:36:50.785567 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0812 02:36:50.785636 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.60:45435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260812 02:36:50.786557 7803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:50.786311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.60:45435 (local address 127.0.0.1:43806)
0812 02:36:50.786435 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0812 02:36:50.786490 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.60:45435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260812 02:36:50.786748 7799 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.103.60:45435: Network error: Client connection negotiation failed: client connection to 127.14.103.60:45435: connect: Connection refused (error 111)
I20260812 02:36:50.805172 7884 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.103.60" port: 45435 } attempt: 1
I20260812 02:36:50.806170 7887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:50.805814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.60:45435 (local address 127.0.0.1:43812)
0812 02:36:50.805991 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0812 02:36:50.806072 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.60:45435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260812 02:36:50.807067 7887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:50.806783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.60:45435 (local address 127.0.0.1:43828)
0812 02:36:50.806912 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0812 02:36:50.806994 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.60:45435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260812 02:36:50.807247 7884 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.103.60:45435: Network error: Client connection negotiation failed: client connection to 127.14.103.60:45435: connect: Connection refused (error 111)
I20260812 02:36:50.861945 7799 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.103.60" port: 45435 } attempt: 2
I20260812 02:36:50.862862 7803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:50.862503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.60:45435 (local address 127.0.0.1:43838)
0812 02:36:50.862683 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0812 02:36:50.862761 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.60:45435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260812 02:36:50.863664 7803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:50.863441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.60:45435 (local address 127.0.0.1:43846)
0812 02:36:50.863556 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0812 02:36:50.863609 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.60:45435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260812 02:36:50.863880 7799 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.103.60:45435: Network error: Client connection negotiation failed: client connection to 127.14.103.60:45435: connect: Connection refused (error 111)
W20260812 02:36:50.870606 7891 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260812 02:36:50.870898 7891 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:36:50.870954 7891 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:36:50.871065 7891 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:36:50.875386 7891 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:36:50.875499 7891 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
I20260812 02:36:50.875413 7884 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.103.60" port: 45435 } attempt: 2
W20260812 02:36:50.875526 7891 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:36:50.875550 7891 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:36:50.875578 7891 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 02:36:50.876278 7887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:50.875934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.60:45435 (local address 127.0.0.1:43848)
0812 02:36:50.876109 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0812 02:36:50.876184 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.60:45435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260812 02:36:50.876986 7887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:50.876781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.60:45435 (local address 127.0.0.1:43860)
0812 02:36:50.876881 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0812 02:36:50.876929 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.60:45435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260812 02:36:50.877141 7884 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.103.60:45435: Network error: Client connection negotiation failed: client connection to 127.14.103.60:45435: connect: Connection refused (error 111)
I20260812 02:36:50.881505 7891 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.103.20:36591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786501898713699-14748-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786501898713699-14748-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.14.103.62:39137,127.14.103.61:35237,127.14.103.60:45435
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.60.keytab
--principal=kudu/127.14.103.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.103.60:45435
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786501898713699-14748-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.14.103.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7891
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786501898713699-14748-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:36:50.882601 7891 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:36:50.883674 7891 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:36:50.892395 7898 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:36:50.892397 7897 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:36:50.892434 7900 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:36:50.892616 7891 server_base.cc:1034] running on GCE node
I20260812 02:36:50.893313 7891 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:36:50.894505 7891 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:36:50.895694 7891 hybrid_clock.cc:648] HybridClock initialized: now 1786502210895676 us; error 61 us; skew 500 ppm
Aug 12 02:36:50 dist-test-slave-2w3w krb5kdc[7709](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502210, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:36:50.898650 7891 init.cc:377] Logged in from keytab as kudu/127.14.103.60@KRBTEST.COM (short username kudu)
I20260812 02:36:50.899644 7891 webserver.cc:460] Webserver started at http://127.14.103.60:37407/ using document root <none> and password file <none>
I20260812 02:36:50.900225 7891 fs_manager.cc:359] Metadata directory not provided
I20260812 02:36:50.900319 7891 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:36:50.900560 7891 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:36:50.902629 7891 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786501898713699-14748-0/minicluster-data/master-2/data/instance:
uuid: "510846af7c5947e58d7014c23dd0c05a"
format_stamp: "Formatted at 2026-08-12 02:36:50 on dist-test-slave-2w3w"
server_key: "84871b9fbcae002695cd40ef36c19944"
server_key_iv: "a3d89f02238f80d3baa8c65dbe613a29"
server_key_version: "encryptionkey@0"
I20260812 02:36:50.903225 7891 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786501898713699-14748-0/minicluster-data/master-2/wal/instance:
uuid: "510846af7c5947e58d7014c23dd0c05a"
format_stamp: "Formatted at 2026-08-12 02:36:50 on dist-test-slave-2w3w"
server_key: "84871b9fbcae002695cd40ef36c19944"
server_key_iv: "a3d89f02238f80d3baa8c65dbe613a29"
server_key_version: "encryptionkey@0"
I20260812 02:36:50.907196 7891 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260812 02:36:50.909818 7908 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:36:50.911355 7891 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 02:36:50.911516 7891 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786501898713699-14748-0/minicluster-data/master-2/data,/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786501898713699-14748-0/minicluster-data/master-2/wal
uuid: "510846af7c5947e58d7014c23dd0c05a"
format_stamp: "Formatted at 2026-08-12 02:36:50 on dist-test-slave-2w3w"
server_key: "84871b9fbcae002695cd40ef36c19944"
server_key_iv: "a3d89f02238f80d3baa8c65dbe613a29"
server_key_version: "encryptionkey@0"
I20260812 02:36:50.911649 7891 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786501898713699-14748-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786501898713699-14748-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786501898713699-14748-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:36:50.927634 7891 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:36:50.930773 7891 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:36:50.931062 7891 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:36:50.938421 7891 rpc_server.cc:307] RPC server started. Bound to: 127.14.103.60:45435
I20260812 02:36:50.938726 7968 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.103.60:45435 every 8 connection(s)
I20260812 02:36:50.939493 7891 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786501898713699-14748-0/minicluster-data/master-2/data/info.pb
I20260812 02:36:50.940651 14748 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbep4M4/build/debug/bin/kudu as pid 7891
I20260812 02:36:50.940798 14748 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786501898713699-14748-0/minicluster-data/master-2/wal/instance
I20260812 02:36:50.941126 14748 external_mini_cluster.cc:1428] Setting key aead31b596842a0cbfe76ac51cebb36e
I20260812 02:36:50.942857 7969 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:36:50.947309 7969 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.103.62" port: 39137 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 02:36:50 dist-test-slave-2w3w krb5kdc[7709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502210, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.103.60@KRBTEST.COM
Aug 12 02:36:50 dist-test-slave-2w3w krb5kdc[7709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502210, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.60@KRBTEST.COM for kudu/127.14.103.62@KRBTEST.COM
I20260812 02:36:50.957355 7972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:50.942542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43870 (local address 127.14.103.60:45435)
0812 02:36:50.942989 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0812 02:36:50.943010 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0812 02:36:50.943056 (+ 46us) server_negotiation.cc:407] Connection header received
0812 02:36:50.943737 (+ 681us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:36:50.943753 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:36:50.944073 (+ 320us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:36:50.944362 (+ 289us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:36:50.945282 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:50.946350 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:50.947602 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:50.948233 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:50.948377 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:36:50.952101 (+ 3724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:36:50.952107 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:36:50.952117 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:36:50.952146 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:36:50.954201 (+ 2055us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:36:50.954735 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:36:50.954753 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:36:50.954781 (+ 28us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:36:50.954908 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:36:50.955316 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:36:50.955320 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:36:50.955322 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:36:50.955741 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:36:50.955961 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0812 02:36:50.956243 (+ 282us) server_negotiation.cc:299] Negotiation successful
0812 02:36:50.956546 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":148,"threads_started":1}
I20260812 02:36:50.960107 7802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:50.949552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39924 (local address 127.14.103.62:39137)
0812 02:36:50.949731 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0812 02:36:50.949736 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:36:50.949993 (+ 257us) server_negotiation.cc:407] Connection header received
0812 02:36:50.950158 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:36:50.950160 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:36:50.950205 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:36:50.950276 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:36:50.951255 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:50.951840 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:50.952760 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:50.952996 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:50.953116 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:36:50.955603 (+ 2487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:36:50.955607 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:36:50.955609 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:36:50.955635 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:36:50.957216 (+ 1581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:36:50.958553 (+ 1337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:36:50.958558 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:36:50.958560 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:36:50.958651 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:36:50.958947 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:36:50.958951 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:36:50.958953 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:36:50.959240 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:36:50.959347 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0812 02:36:50.959783 (+ 436us) server_negotiation.cc:299] Negotiation successful
0812 02:36:50.959944 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260812 02:36:50.960119 7973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:50.949352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:39137 (local address 127.0.0.1:39924)
0812 02:36:50.949819 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0812 02:36:50.949893 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0812 02:36:50.950051 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:36:50.950379 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:36:50.950387 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:36:50.950920 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:36:50.951118 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:36:50.951125 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:36:50.951966 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:36:50.951969 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:36:50.952625 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:36:50.952634 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:36:50.953118 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:36:50.953121 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:36:50.953268 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:36:50.953749 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:36:50.953761 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:36:50.955441 (+ 1680us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:36:50.957336 (+ 1895us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:36:50.957347 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:36:50.957354 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:36:50.958437 (+ 1083us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:36:50.958752 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:36:50.958757 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:36:50.958758 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:36:50.958850 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:36:50.959388 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:36:50.959394 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:36:50.959530 (+ 136us) client_negotiation.cc:769] Sending connection context
0812 02:36:50.959701 (+ 171us) client_negotiation.cc:240] Negotiation successful
0812 02:36:50.959917 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":141,"threads_started":1}
I20260812 02:36:50.961169 7969 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.103.61" port: 35237 } has no permanent_uuid. Determining permanent_uuid...
I20260812 02:36:50.963048 7799 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.103.60" port: 45435 } attempt: 3
Aug 12 02:36:50 dist-test-slave-2w3w krb5kdc[7709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502210, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.103.62@KRBTEST.COM
Aug 12 02:36:50 dist-test-slave-2w3w krb5kdc[7709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502210, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.60@KRBTEST.COM for kudu/127.14.103.61@KRBTEST.COM
I20260812 02:36:50.968322 7884 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.103.60" port: 45435 } attempt: 3
Aug 12 02:36:50 dist-test-slave-2w3w krb5kdc[7709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502210, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.62@KRBTEST.COM for kudu/127.14.103.60@KRBTEST.COM
I20260812 02:36:50.972256 7974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:50.959778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39930 (local address 127.14.103.62:39137)
0812 02:36:50.960183 (+ 405us) server_negotiation.cc:206] Beginning negotiation
0812 02:36:50.960264 (+ 81us) server_negotiation.cc:399] Waiting for connection header
0812 02:36:50.960279 (+ 15us) server_negotiation.cc:407] Connection header received
0812 02:36:50.960321 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:36:50.960325 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:36:50.960388 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:36:50.960467 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:36:50.961203 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:50.961834 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:50.962679 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:50.962949 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:50.963093 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:36:50.966319 (+ 3226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:36:50.966323 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:36:50.966326 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:36:50.966359 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:36:50.968504 (+ 2145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:36:50.969161 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:36:50.969167 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:36:50.969169 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:36:50.969270 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:36:50.969590 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:36:50.969596 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:36:50.969600 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:36:50.970795 (+ 1195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:36:50.971759 (+ 964us) server_negotiation.cc:1035] Waiting for connection context
0812 02:36:50.971841 (+ 82us) server_negotiation.cc:299] Negotiation successful
0812 02:36:50.972008 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":131,"threads_started":1}
I20260812 02:36:50.972515 7890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:50.961845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40352 (local address 127.14.103.61:35237)
0812 02:36:50.961972 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0812 02:36:50.961978 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 02:36:50.962009 (+ 31us) server_negotiation.cc:407] Connection header received
0812 02:36:50.962051 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:36:50.962054 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:36:50.962109 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:36:50.962197 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:36:50.962632 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:50.963305 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:50.964479 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:50.964832 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:50.964952 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:36:50.968186 (+ 3234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:36:50.968192 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:36:50.968197 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:36:50.968229 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:36:50.970426 (+ 2197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:36:50.971002 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:36:50.971005 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:36:50.971007 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:36:50.971100 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:36:50.971454 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:36:50.971457 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:36:50.971459 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:36:50.971689 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:36:50.971757 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0812 02:36:50.972221 (+ 464us) server_negotiation.cc:299] Negotiation successful
0812 02:36:50.972357 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260812 02:36:50.973471 7973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:50.961727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.61:35237 (local address 127.0.0.1:40352)
0812 02:36:50.961854 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0812 02:36:50.961868 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 02:36:50.961945 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:36:50.962211 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:36:50.962214 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:36:50.962402 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:36:50.962504 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:36:50.962509 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:36:50.963429 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:36:50.963435 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:36:50.964330 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:36:50.964342 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:36:50.964952 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:36:50.964955 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:36:50.965074 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:36:50.966428 (+ 1354us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:36:50.966443 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:36:50.968013 (+ 1570us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:36:50.970542 (+ 2529us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:36:50.970547 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:36:50.970552 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:36:50.970858 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:36:50.971227 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:36:50.971231 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:36:50.971235 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:36:50.971325 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:36:50.971836 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:36:50.971839 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:36:50.971931 (+ 92us) client_negotiation.cc:769] Sending connection context
0812 02:36:50.973139 (+ 1208us) client_negotiation.cc:240] Negotiation successful
0812 02:36:50.973283 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260812 02:36:50.973932 7803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:50.963792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.60:45435 (local address 127.0.0.1:43872)
0812 02:36:50.963918 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0812 02:36:50.963935 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 02:36:50.964049 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:36:50.964302 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:36:50.964304 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:36:50.964502 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:36:50.964593 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:36:50.964599 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:36:50.965811 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:36:50.965814 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:36:50.966581 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:36:50.966594 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:36:50.967295 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:36:50.967299 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:36:50.967409 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:36:50.967857 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:36:50.967872 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:36:50.969384 (+ 1512us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:36:50.972196 (+ 2812us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:36:50.972203 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:36:50.972208 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:36:50.972461 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:36:50.972795 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:36:50.972800 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:36:50.972804 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:36:50.972919 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:36:50.973388 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:36:50.973391 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:36:50.973491 (+ 100us) client_negotiation.cc:769] Sending connection context
0812 02:36:50.973611 (+ 120us) client_negotiation.cc:240] Negotiation successful
0812 02:36:50.973765 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260812 02:36:50.974439 7972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:50.963588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43872 (local address 127.14.103.60:45435)
0812 02:36:50.963715 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0812 02:36:50.963719 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:36:50.964012 (+ 293us) server_negotiation.cc:407] Connection header received
0812 02:36:50.964150 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:36:50.964153 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:36:50.964193 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:36:50.964286 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:36:50.964717 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:50.965636 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:50.966701 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:50.967144 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:50.967650 (+ 506us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:36:50.969520 (+ 1870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:36:50.969527 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:36:50.969530 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:36:50.969555 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:36:50.972063 (+ 2508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:36:50.972559 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:36:50.972563 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:36:50.972564 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:36:50.972658 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:36:50.973027 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:36:50.973030 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:36:50.973031 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:36:50.973245 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:36:50.973340 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0812 02:36:50.974122 (+ 782us) server_negotiation.cc:299] Negotiation successful
0812 02:36:50.974254 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260812 02:36:50.975168 7969 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.103.60" port: 45435 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 02:36:50 dist-test-slave-2w3w krb5kdc[7709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502210, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.61@KRBTEST.COM for kudu/127.14.103.60@KRBTEST.COM
I20260812 02:36:50.978567 7799 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d4628539e294e919d4e53bea3a836c9: Bootstrap starting.
I20260812 02:36:50.981032 7887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:50.968772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.60:45435 (local address 127.0.0.1:43874)
0812 02:36:50.969091 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0812 02:36:50.969107 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 02:36:50.969185 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:36:50.970525 (+ 1340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:36:50.970531 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:36:50.970727 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:36:50.970855 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:36:50.970862 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:36:50.972246 (+ 1384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:36:50.972253 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:36:50.973133 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:36:50.973148 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:36:50.973996 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:36:50.974001 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:36:50.974103 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:36:50.975561 (+ 1458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:36:50.975577 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:36:50.977168 (+ 1591us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:36:50.979326 (+ 2158us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:36:50.979330 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:36:50.979332 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:36:50.979605 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:36:50.979942 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:36:50.979945 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:36:50.979947 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:36:50.980048 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:36:50.980539 (+ 491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:36:50.980542 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:36:50.980640 (+ 98us) client_negotiation.cc:769] Sending connection context
0812 02:36:50.980721 (+ 81us) client_negotiation.cc:240] Negotiation successful
0812 02:36:50.980869 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":218}
I20260812 02:36:50.981750 7975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:50.969239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43874 (local address 127.14.103.60:45435)
0812 02:36:50.969988 (+ 749us) server_negotiation.cc:206] Beginning negotiation
0812 02:36:50.970000 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 02:36:50.970014 (+ 14us) server_negotiation.cc:407] Connection header received
0812 02:36:50.970049 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:36:50.970052 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:36:50.970097 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:36:50.970197 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:36:50.971033 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:50.972063 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:50.973363 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:50.973801 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:50.973960 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:36:50.977341 (+ 3381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:36:50.977345 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:36:50.977347 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:36:50.977385 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:36:50.979190 (+ 1805us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:36:50.979734 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:36:50.979737 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:36:50.979739 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:36:50.979831 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:36:50.980175 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:36:50.980178 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:36:50.980180 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:36:50.980420 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:36:50.981254 (+ 834us) server_negotiation.cc:1035] Waiting for connection context
0812 02:36:50.981435 (+ 181us) server_negotiation.cc:299] Negotiation successful
0812 02:36:50.981571 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":665,"spinlock_wait_cycles":768,"thread_start_us":154,"threads_started":1}
Aug 12 02:36:50 dist-test-slave-2w3w krb5kdc[7709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502210, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.60@KRBTEST.COM for kudu/127.14.103.60@KRBTEST.COM
Aug 12 02:36:50 dist-test-slave-2w3w krb5kdc[7709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502210, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.103.61@KRBTEST.COM
I20260812 02:36:50.986238 7884 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d52e652d44204f9aad62c52c10a3f0fd: Bootstrap starting.
I20260812 02:36:50.988469 7890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:50.974825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40368 (local address 127.14.103.61:35237)
0812 02:36:50.975087 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0812 02:36:50.975091 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:36:50.975403 (+ 312us) server_negotiation.cc:407] Connection header received
0812 02:36:50.976386 (+ 983us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:36:50.976392 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:36:50.976447 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:36:50.976496 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:36:50.977661 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:50.978453 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:50.979775 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:50.980288 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:50.980362 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:36:50.984003 (+ 3641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:36:50.984007 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:36:50.984013 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:36:50.984048 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:36:50.986269 (+ 2221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:36:50.986778 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:36:50.986804 (+ 26us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:36:50.986807 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:36:50.986912 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:36:50.987333 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:36:50.987340 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:36:50.987344 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:36:50.987636 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:36:50.987805 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0812 02:36:50.988145 (+ 340us) server_negotiation.cc:299] Negotiation successful
0812 02:36:50.988276 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178}
I20260812 02:36:50.989269 7972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:50.975886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43878 (local address 127.14.103.60:45435)
0812 02:36:50.976016 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0812 02:36:50.976020 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:36:50.976184 (+ 164us) server_negotiation.cc:407] Connection header received
0812 02:36:50.976322 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:36:50.976326 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:36:50.976370 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:36:50.976446 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:36:50.977111 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:50.978569 (+ 1458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:50.979758 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:50.980265 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:50.980362 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:36:50.983510 (+ 3148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:36:50.983515 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:36:50.983517 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:36:50.983545 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:36:50.986175 (+ 2630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:36:50.987074 (+ 899us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:36:50.987077 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:36:50.987079 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:36:50.987167 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:36:50.987460 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:36:50.987463 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:36:50.987464 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:36:50.987673 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:36:50.987759 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0812 02:36:50.988781 (+ 1022us) server_negotiation.cc:299] Negotiation successful
0812 02:36:50.989093 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260812 02:36:50.989923 7973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:50.975751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.60:45435 (local address 127.0.0.1:43878)
0812 02:36:50.976094 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0812 02:36:50.976117 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0812 02:36:50.976226 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:36:50.976554 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:36:50.976558 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:36:50.976765 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:36:50.976879 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:36:50.976887 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:36:50.978680 (+ 1793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:36:50.978683 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:36:50.979629 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:36:50.979640 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:36:50.980526 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:36:50.980530 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:36:50.980610 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:36:50.981301 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:36:50.981330 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:36:50.983351 (+ 2021us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:36:50.986341 (+ 2990us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:36:50.986345 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:36:50.986347 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:36:50.986890 (+ 543us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:36:50.987263 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:36:50.987266 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:36:50.987268 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:36:50.987349 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:36:50.987792 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:36:50.987794 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:36:50.987870 (+ 76us) client_negotiation.cc:769] Sending connection context
0812 02:36:50.987959 (+ 89us) client_negotiation.cc:240] Negotiation successful
0812 02:36:50.988113 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"spinlock_wait_cycles":2432}
I20260812 02:36:50.991242 7884 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d52e652d44204f9aad62c52c10a3f0fd: Neither blocks nor log segments found. Creating new log.
I20260812 02:36:50.991412 7799 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6d4628539e294e919d4e53bea3a836c9: Neither blocks nor log segments found. Creating new log.
I20260812 02:36:50.992213 7884 log.cc:824] T 00000000000000000000000000000000 P d52e652d44204f9aad62c52c10a3f0fd: Log is configured to *not* fsync() on all Append() calls
I20260812 02:36:50.992326 7799 log.cc:824] T 00000000000000000000000000000000 P 6d4628539e294e919d4e53bea3a836c9: Log is configured to *not* fsync() on all Append() calls
I20260812 02:36:50.992870 14748 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 02:36:50.993374 7969 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 510846af7c5947e58d7014c23dd0c05a: Bootstrap starting.
I20260812 02:36:50.994848 7884 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d52e652d44204f9aad62c52c10a3f0fd: No bootstrap required, opened a new log
I20260812 02:36:50.994415 7766 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "6d4628539e294e919d4e53bea3a836c9"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39930
I20260812 02:36:50.996172 7969 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 510846af7c5947e58d7014c23dd0c05a: Neither blocks nor log segments found. Creating new log.
I20260812 02:36:50.996953 7799 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d4628539e294e919d4e53bea3a836c9: No bootstrap required, opened a new log
I20260812 02:36:50.997063 7969 log.cc:824] T 00000000000000000000000000000000 P 510846af7c5947e58d7014c23dd0c05a: Log is configured to *not* fsync() on all Append() calls
I20260812 02:36:50.999238 7884 raft_consensus.cc:348] T 00000000000000000000000000000000 P d52e652d44204f9aad62c52c10a3f0fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4628539e294e919d4e53bea3a836c9" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 39137 } } peers { permanent_uuid: "d52e652d44204f9aad62c52c10a3f0fd" member_type: VOTER last_known_addr { host: "127.14.103.61" port: 35237 } } peers { permanent_uuid: "510846af7c5947e58d7014c23dd0c05a" member_type: VOTER last_known_addr { host: "127.14.103.60" port: 45435 } }
I20260812 02:36:50.999534 7884 raft_consensus.cc:729] T 00000000000000000000000000000000 P d52e652d44204f9aad62c52c10a3f0fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d52e652d44204f9aad62c52c10a3f0fd, State: Initialized, Role: FOLLOWER
I20260812 02:36:50.999538 7799 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6d4628539e294e919d4e53bea3a836c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4628539e294e919d4e53bea3a836c9" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 39137 } } peers { permanent_uuid: "d52e652d44204f9aad62c52c10a3f0fd" member_type: VOTER last_known_addr { host: "127.14.103.61" port: 35237 } } peers { permanent_uuid: "510846af7c5947e58d7014c23dd0c05a" member_type: VOTER last_known_addr { host: "127.14.103.60" port: 45435 } }
I20260812 02:36:50.999759 7799 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6d4628539e294e919d4e53bea3a836c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d4628539e294e919d4e53bea3a836c9, State: Initialized, Role: FOLLOWER
I20260812 02:36:50.999899 7884 consensus_queue.cc:260] T 00000000000000000000000000000000 P d52e652d44204f9aad62c52c10a3f0fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4628539e294e919d4e53bea3a836c9" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 39137 } } peers { permanent_uuid: "d52e652d44204f9aad62c52c10a3f0fd" member_type: VOTER last_known_addr { host: "127.14.103.61" port: 35237 } } peers { permanent_uuid: "510846af7c5947e58d7014c23dd0c05a" member_type: VOTER last_known_addr { host: "127.14.103.60" port: 45435 } }
I20260812 02:36:51.000073 7799 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6d4628539e294e919d4e53bea3a836c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4628539e294e919d4e53bea3a836c9" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 39137 } } peers { permanent_uuid: "d52e652d44204f9aad62c52c10a3f0fd" member_type: VOTER last_known_addr { host: "127.14.103.61" port: 35237 } } peers { permanent_uuid: "510846af7c5947e58d7014c23dd0c05a" member_type: VOTER last_known_addr { host: "127.14.103.60" port: 45435 } }
I20260812 02:36:51.000463 7969 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 510846af7c5947e58d7014c23dd0c05a: No bootstrap required, opened a new log
I20260812 02:36:51.000629 7980 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d4628539e294e919d4e53bea3a836c9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4628539e294e919d4e53bea3a836c9" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 39137 } } peers { permanent_uuid: "d52e652d44204f9aad62c52c10a3f0fd" member_type: VOTER last_known_addr { host: "127.14.103.61" port: 35237 } } peers { permanent_uuid: "510846af7c5947e58d7014c23dd0c05a" member_type: VOTER last_known_addr { host: "127.14.103.60" port: 45435 } } }
I20260812 02:36:51.000829 7980 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d4628539e294e919d4e53bea3a836c9 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 02:36:51.001345 7799 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6d4628539e294e919d4e53bea3a836c9 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:36:51.001401 7884 sys_catalog.cc:564] T 00000000000000000000000000000000 P d52e652d44204f9aad62c52c10a3f0fd [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:36:51.001461 7766 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6d4628539e294e919d4e53bea3a836c9 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260812 02:36:51.001588 7766 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6d4628539e294e919d4e53bea3a836c9 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:36:51.003211 7766 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6d4628539e294e919d4e53bea3a836c9 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4628539e294e919d4e53bea3a836c9" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 39137 } } peers { permanent_uuid: "d52e652d44204f9aad62c52c10a3f0fd" member_type: VOTER last_known_addr { host: "127.14.103.61" port: 35237 } } peers { permanent_uuid: "510846af7c5947e58d7014c23dd0c05a" member_type: VOTER last_known_addr { host: "127.14.103.60" port: 45435 } }
I20260812 02:36:51.004776 7766 leader_election.cc:288] T 00000000000000000000000000000000 P 6d4628539e294e919d4e53bea3a836c9 [CANDIDATE]: Term 1 election: Requested vote from peers d52e652d44204f9aad62c52c10a3f0fd (127.14.103.61:35237), 510846af7c5947e58d7014c23dd0c05a (127.14.103.60:45435)
I20260812 02:36:51.004997 7854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6d4628539e294e919d4e53bea3a836c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "d52e652d44204f9aad62c52c10a3f0fd"
I20260812 02:36:51.005230 7854 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d52e652d44204f9aad62c52c10a3f0fd [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:36:51.006803 7854 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d52e652d44204f9aad62c52c10a3f0fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4628539e294e919d4e53bea3a836c9 in term 1.
W20260812 02:36:51.007596 8002 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d52e652d44204f9aad62c52c10a3f0fd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 02:36:51.007676 8002 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 02:36:51.008411 7742 leader_election.cc:302] T 00000000000000000000000000000000 P 6d4628539e294e919d4e53bea3a836c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4628539e294e919d4e53bea3a836c9, d52e652d44204f9aad62c52c10a3f0fd; no voters:
I20260812 02:36:51.008726 7980 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6d4628539e294e919d4e53bea3a836c9 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:36:51.008908 7980 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6d4628539e294e919d4e53bea3a836c9 [term 1 LEADER]: Becoming Leader. State: Replica: 6d4628539e294e919d4e53bea3a836c9, State: Running, Role: LEADER
I20260812 02:36:51.009231 7980 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6d4628539e294e919d4e53bea3a836c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4628539e294e919d4e53bea3a836c9" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 39137 } } peers { permanent_uuid: "d52e652d44204f9aad62c52c10a3f0fd" member_type: VOTER last_known_addr { host: "127.14.103.61" port: 35237 } } peers { permanent_uuid: "510846af7c5947e58d7014c23dd0c05a" member_type: VOTER last_known_addr { host: "127.14.103.60" port: 45435 } }
W20260812 02:36:51.009510 8003 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6d4628539e294e919d4e53bea3a836c9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 02:36:51.009588 8003 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 02:36:51.009373 7969 raft_consensus.cc:348] T 00000000000000000000000000000000 P 510846af7c5947e58d7014c23dd0c05a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4628539e294e919d4e53bea3a836c9" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 39137 } } peers { permanent_uuid: "d52e652d44204f9aad62c52c10a3f0fd" member_type: VOTER last_known_addr { host: "127.14.103.61" port: 35237 } } peers { permanent_uuid: "510846af7c5947e58d7014c23dd0c05a" member_type: VOTER last_known_addr { host: "127.14.103.60" port: 45435 } }
I20260812 02:36:51.009652 7969 raft_consensus.cc:729] T 00000000000000000000000000000000 P 510846af7c5947e58d7014c23dd0c05a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 510846af7c5947e58d7014c23dd0c05a, State: Initialized, Role: FOLLOWER
I20260812 02:36:51.010004 7969 consensus_queue.cc:260] T 00000000000000000000000000000000 P 510846af7c5947e58d7014c23dd0c05a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4628539e294e919d4e53bea3a836c9" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 39137 } } peers { permanent_uuid: "d52e652d44204f9aad62c52c10a3f0fd" member_type: VOTER last_known_addr { host: "127.14.103.61" port: 35237 } } peers { permanent_uuid: "510846af7c5947e58d7014c23dd0c05a" member_type: VOTER last_known_addr { host: "127.14.103.60" port: 45435 } }
I20260812 02:36:51.011646 7969 sys_catalog.cc:564] T 00000000000000000000000000000000 P 510846af7c5947e58d7014c23dd0c05a [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:36:51.012208 7938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6d4628539e294e919d4e53bea3a836c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "510846af7c5947e58d7014c23dd0c05a"
I20260812 02:36:51.012436 7938 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 510846af7c5947e58d7014c23dd0c05a [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:36:51.013948 7938 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 510846af7c5947e58d7014c23dd0c05a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4628539e294e919d4e53bea3a836c9 in term 1.
W20260812 02:36:51.018364 8029 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 510846af7c5947e58d7014c23dd0c05a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 02:36:51.018465 8029 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 02:36:51.018755 7980 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d4628539e294e919d4e53bea3a836c9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6d4628539e294e919d4e53bea3a836c9. Latest consensus state: current_term: 1 leader_uuid: "6d4628539e294e919d4e53bea3a836c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4628539e294e919d4e53bea3a836c9" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 39137 } } peers { permanent_uuid: "d52e652d44204f9aad62c52c10a3f0fd" member_type: VOTER last_known_addr { host: "127.14.103.61" port: 35237 } } peers { permanent_uuid: "510846af7c5947e58d7014c23dd0c05a" member_type: VOTER last_known_addr { host: "127.14.103.60" port: 45435 } } }
I20260812 02:36:51.018890 7980 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d4628539e294e919d4e53bea3a836c9 [sys.catalog]: This master's current role is: LEADER
I20260812 02:36:51.003729 7979 sys_catalog.cc:455] T 00000000000000000000000000000000 P d52e652d44204f9aad62c52c10a3f0fd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4628539e294e919d4e53bea3a836c9" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 39137 } } peers { permanent_uuid: "d52e652d44204f9aad62c52c10a3f0fd" member_type: VOTER last_known_addr { host: "127.14.103.61" port: 35237 } } peers { permanent_uuid: "510846af7c5947e58d7014c23dd0c05a" member_type: VOTER last_known_addr { host: "127.14.103.60" port: 45435 } } }
I20260812 02:36:51.019320 7979 sys_catalog.cc:458] T 00000000000000000000000000000000 P d52e652d44204f9aad62c52c10a3f0fd [sys.catalog]: This master's current role is: FOLLOWER
I20260812 02:36:51.021116 8033 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 02:36:51.022109 7722 master_runner.cc:263] No leader master found from master 6d4628539e294e919d4e53bea3a836c9
I20260812 02:36:51.022182 7722 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260812 02:36:51.024474 7807 master_runner.cc:263] No leader master found from master d52e652d44204f9aad62c52c10a3f0fd
I20260812 02:36:51.024545 7807 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260812 02:36:51.024533 8033 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 02:36:51.027139 8015 sys_catalog.cc:455] T 00000000000000000000000000000000 P 510846af7c5947e58d7014c23dd0c05a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4628539e294e919d4e53bea3a836c9" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 39137 } } peers { permanent_uuid: "d52e652d44204f9aad62c52c10a3f0fd" member_type: VOTER last_known_addr { host: "127.14.103.61" port: 35237 } } peers { permanent_uuid: "510846af7c5947e58d7014c23dd0c05a" member_type: VOTER last_known_addr { host: "127.14.103.60" port: 45435 } } }
I20260812 02:36:51.027277 8015 sys_catalog.cc:458] T 00000000000000000000000000000000 P 510846af7c5947e58d7014c23dd0c05a [sys.catalog]: This master's current role is: FOLLOWER
I20260812 02:36:51.030577 7938 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 510846af7c5947e58d7014c23dd0c05a [term 1 FOLLOWER]: Refusing update from remote peer 6d4628539e294e919d4e53bea3a836c9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:36:51.030578 7854 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d52e652d44204f9aad62c52c10a3f0fd [term 1 FOLLOWER]: Refusing update from remote peer 6d4628539e294e919d4e53bea3a836c9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:36:51.031373 7980 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6d4628539e294e919d4e53bea3a836c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d52e652d44204f9aad62c52c10a3f0fd" member_type: VOTER last_known_addr { host: "127.14.103.61" port: 35237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:36:51.031829 7980 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6d4628539e294e919d4e53bea3a836c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "510846af7c5947e58d7014c23dd0c05a" member_type: VOTER last_known_addr { host: "127.14.103.60" port: 45435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:36:51.035804 7979 sys_catalog.cc:455] T 00000000000000000000000000000000 P d52e652d44204f9aad62c52c10a3f0fd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6d4628539e294e919d4e53bea3a836c9. Latest consensus state: current_term: 1 leader_uuid: "6d4628539e294e919d4e53bea3a836c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4628539e294e919d4e53bea3a836c9" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 39137 } } peers { permanent_uuid: "d52e652d44204f9aad62c52c10a3f0fd" member_type: VOTER last_known_addr { host: "127.14.103.61" port: 35237 } } peers { permanent_uuid: "510846af7c5947e58d7014c23dd0c05a" member_type: VOTER last_known_addr { host: "127.14.103.60" port: 45435 } } }
I20260812 02:36:51.035964 7979 sys_catalog.cc:458] T 00000000000000000000000000000000 P d52e652d44204f9aad62c52c10a3f0fd [sys.catalog]: This master's current role is: FOLLOWER
I20260812 02:36:51.036267 8015 sys_catalog.cc:455] T 00000000000000000000000000000000 P 510846af7c5947e58d7014c23dd0c05a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6d4628539e294e919d4e53bea3a836c9. Latest consensus state: current_term: 1 leader_uuid: "6d4628539e294e919d4e53bea3a836c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4628539e294e919d4e53bea3a836c9" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 39137 } } peers { permanent_uuid: "d52e652d44204f9aad62c52c10a3f0fd" member_type: VOTER last_known_addr { host: "127.14.103.61" port: 35237 } } peers { permanent_uuid: "510846af7c5947e58d7014c23dd0c05a" member_type: VOTER last_known_addr { host: "127.14.103.60" port: 45435 } } }
I20260812 02:36:51.036437 8015 sys_catalog.cc:458] T 00000000000000000000000000000000 P 510846af7c5947e58d7014c23dd0c05a [sys.catalog]: This master's current role is: FOLLOWER
I20260812 02:36:51.045074 7972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:51.010151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43884 (local address 127.14.103.60:45435)
0812 02:36:51.010279 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0812 02:36:51.010284 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:36:51.022769 (+ 12485us) server_negotiation.cc:407] Connection header received
0812 02:36:51.026743 (+ 3974us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:36:51.026747 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:36:51.026800 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:36:51.026858 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:36:51.036169 (+ 9311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:51.037009 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:51.038331 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:51.038678 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:51.038805 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:36:51.040887 (+ 2082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:36:51.040891 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:36:51.040894 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:36:51.040940 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:36:51.043049 (+ 2109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:36:51.043596 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:36:51.043600 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:36:51.043602 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:36:51.043692 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:36:51.043994 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:36:51.043997 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:36:51.043999 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:36:51.044210 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:36:51.044318 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0812 02:36:51.044688 (+ 370us) server_negotiation.cc:299] Negotiation successful
0812 02:36:51.044839 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260812 02:36:51.048782 7980 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d4628539e294e919d4e53bea3a836c9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6d4628539e294e919d4e53bea3a836c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4628539e294e919d4e53bea3a836c9" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 39137 } } peers { permanent_uuid: "d52e652d44204f9aad62c52c10a3f0fd" member_type: VOTER last_known_addr { host: "127.14.103.61" port: 35237 } } peers { permanent_uuid: "510846af7c5947e58d7014c23dd0c05a" member_type: VOTER last_known_addr { host: "127.14.103.60" port: 45435 } } }
I20260812 02:36:51.048930 7980 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d4628539e294e919d4e53bea3a836c9 [sys.catalog]: This master's current role is: LEADER
I20260812 02:36:51.049789 7890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:51.011154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40370 (local address 127.14.103.61:35237)
0812 02:36:51.011286 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0812 02:36:51.011292 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 02:36:51.028701 (+ 17409us) server_negotiation.cc:407] Connection header received
0812 02:36:51.028872 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:36:51.028875 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:36:51.028924 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:36:51.028975 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:36:51.038528 (+ 9553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:51.039313 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:51.040506 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:51.040840 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:51.040915 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:36:51.045389 (+ 4474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:36:51.045393 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:36:51.045396 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:36:51.045440 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:36:51.047567 (+ 2127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:36:51.048149 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:36:51.048156 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:36:51.048159 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:36:51.048263 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:36:51.048602 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:36:51.048606 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:36:51.048609 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:36:51.048858 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:36:51.048990 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0812 02:36:51.049482 (+ 492us) server_negotiation.cc:299] Negotiation successful
0812 02:36:51.049623 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260812 02:36:51.050578 8015 sys_catalog.cc:455] T 00000000000000000000000000000000 P 510846af7c5947e58d7014c23dd0c05a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6d4628539e294e919d4e53bea3a836c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4628539e294e919d4e53bea3a836c9" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 39137 } } peers { permanent_uuid: "d52e652d44204f9aad62c52c10a3f0fd" member_type: VOTER last_known_addr { host: "127.14.103.61" port: 35237 } } peers { permanent_uuid: "510846af7c5947e58d7014c23dd0c05a" member_type: VOTER last_known_addr { host: "127.14.103.60" port: 45435 } } }
I20260812 02:36:51.050719 8015 sys_catalog.cc:458] T 00000000000000000000000000000000 P 510846af7c5947e58d7014c23dd0c05a [sys.catalog]: This master's current role is: FOLLOWER
I20260812 02:36:51.051141 8034 mvcc.cc:205] Tried to move back new op lower bound from 7317513056373407744 to 7317513056297459712. Current Snapshot: MvccSnapshot[applied={T|T < 7317513056373407744}]
I20260812 02:36:51.051514 8035 mvcc.cc:205] Tried to move back new op lower bound from 7317513056373407744 to 7317513056297459712. Current Snapshot: MvccSnapshot[applied={T|T < 7317513056373407744}]
I20260812 02:36:51.055401 8033 catalog_manager.cc:1347] Generated new cluster ID: 4b974d3fbed541d6b9f59e0df6d3002f
I20260812 02:36:51.055495 8033 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 02:36:51.063819 7974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:51.017121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39938 (local address 127.14.103.62:39137)
0812 02:36:51.017266 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0812 02:36:51.017270 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:36:51.041621 (+ 24351us) server_negotiation.cc:407] Connection header received
0812 02:36:51.041753 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:36:51.041757 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:36:51.041825 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:36:51.041908 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:36:51.054649 (+ 12741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:51.055524 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:51.056775 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:51.057147 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:51.057233 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:36:51.059632 (+ 2399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:36:51.059636 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:36:51.059639 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:36:51.059674 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:36:51.061679 (+ 2005us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:36:51.062254 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:36:51.062258 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:36:51.062260 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:36:51.062372 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:36:51.062705 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:36:51.062709 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:36:51.062711 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:36:51.062979 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:36:51.063119 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0812 02:36:51.063502 (+ 383us) server_negotiation.cc:299] Negotiation successful
0812 02:36:51.063645 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260812 02:36:51.083523 8040 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d4628539e294e919d4e53bea3a836c9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6d4628539e294e919d4e53bea3a836c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4628539e294e919d4e53bea3a836c9" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 39137 } } peers { permanent_uuid: "d52e652d44204f9aad62c52c10a3f0fd" member_type: VOTER last_known_addr { host: "127.14.103.61" port: 35237 } } peers { permanent_uuid: "510846af7c5947e58d7014c23dd0c05a" member_type: VOTER last_known_addr { host: "127.14.103.60" port: 45435 } } }
I20260812 02:36:51.083693 8040 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d4628539e294e919d4e53bea3a836c9 [sys.catalog]: This master's current role is: LEADER
I20260812 02:36:51.083798 8033 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 02:36:51.085079 8033 catalog_manager.cc:1504] Loading token signing keys...
I20260812 02:36:51.086705 7979 sys_catalog.cc:455] T 00000000000000000000000000000000 P d52e652d44204f9aad62c52c10a3f0fd [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6d4628539e294e919d4e53bea3a836c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4628539e294e919d4e53bea3a836c9" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 39137 } } peers { permanent_uuid: "d52e652d44204f9aad62c52c10a3f0fd" member_type: VOTER last_known_addr { host: "127.14.103.61" port: 35237 } } peers { permanent_uuid: "510846af7c5947e58d7014c23dd0c05a" member_type: VOTER last_known_addr { host: "127.14.103.60" port: 45435 } } }
I20260812 02:36:51.086849 7979 sys_catalog.cc:458] T 00000000000000000000000000000000 P d52e652d44204f9aad62c52c10a3f0fd [sys.catalog]: This master's current role is: FOLLOWER
I20260812 02:36:51.088861 8036 mvcc.cc:205] Tried to move back new op lower bound from 7317513056373407744 to 7317513056297459712. Current Snapshot: MvccSnapshot[applied={T|T < 7317513056373407744}]
I20260812 02:36:51.104028 8033 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6d4628539e294e919d4e53bea3a836c9: Generated new TSK 0
I20260812 02:36:51.105085 8033 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260812 02:36:51.138661 7972 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:36:51.136135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43894 (local address 127.14.103.60:45435)
0812 02:36:51.136297 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0812 02:36:51.136302 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:36:51.137538 (+ 1236us) server_negotiation.cc:407] Connection header received
0812 02:36:51.137738 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:36:51.137742 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:36:51.137791 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:36:51.137845 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:36:51.138515 (+ 670us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43894: BlockingRecv error: recv got EOF from 127.0.0.1:43894 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260812 02:36:51.140259 7890 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:36:51.136885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40384 (local address 127.14.103.61:35237)
0812 02:36:51.137008 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0812 02:36:51.137012 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:36:51.139209 (+ 2197us) server_negotiation.cc:407] Connection header received
0812 02:36:51.139448 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:36:51.139452 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:36:51.139502 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:36:51.139558 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:36:51.140135 (+ 577us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40384: BlockingRecv error: recv got EOF from 127.0.0.1:40384 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260812 02:36:51.142204 7974 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:36:51.140783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39948 (local address 127.14.103.62:39137)
0812 02:36:51.140932 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0812 02:36:51.140937 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:36:51.141288 (+ 351us) server_negotiation.cc:407] Connection header received
0812 02:36:51.141474 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:36:51.141478 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:36:51.141528 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:36:51.141580 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:36:51.142076 (+ 496us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39948: BlockingRecv error: recv got EOF from 127.0.0.1:39948 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260812 02:36:51.146904 7890 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:36:51.145262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40386 (local address 127.14.103.61:35237)
0812 02:36:51.145370 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0812 02:36:51.145374 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:36:51.145862 (+ 488us) server_negotiation.cc:407] Connection header received
0812 02:36:51.146105 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:36:51.146109 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:36:51.146193 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:36:51.146251 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:36:51.146794 (+ 543us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40386: BlockingRecv error: recv got EOF from 127.0.0.1:40386 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260812 02:36:51.152300 7890 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:36:51.150637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40392 (local address 127.14.103.61:35237)
0812 02:36:51.150747 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0812 02:36:51.150752 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:36:51.151337 (+ 585us) server_negotiation.cc:407] Connection header received
0812 02:36:51.151541 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:36:51.151544 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:36:51.151598 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:36:51.151650 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:36:51.152193 (+ 543us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40392: BlockingRecv error: recv got EOF from 127.0.0.1:40392 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260812 02:36:51.162577 7890 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:36:51.156852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40406 (local address 127.14.103.61:35237)
0812 02:36:51.156985 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0812 02:36:51.156989 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:36:51.161372 (+ 4383us) server_negotiation.cc:407] Connection header received
0812 02:36:51.161644 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:36:51.161648 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:36:51.161713 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:36:51.161773 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:36:51.162448 (+ 675us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40406: BlockingRecv error: recv got EOF from 127.0.0.1:40406 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260812 02:36:51.171793 7890 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:36:51.169716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40410 (local address 127.14.103.61:35237)
0812 02:36:51.169890 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0812 02:36:51.169895 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:36:51.170559 (+ 664us) server_negotiation.cc:407] Connection header received
0812 02:36:51.170790 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:36:51.170795 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:36:51.170863 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:36:51.170947 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:36:51.171661 (+ 714us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40410: BlockingRecv error: recv got EOF from 127.0.0.1:40410 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
W20260812 02:36:51.184854 7890 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:36:51.182765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40412 (local address 127.14.103.61:35237)
0812 02:36:51.182904 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0812 02:36:51.182908 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:36:51.183689 (+ 781us) server_negotiation.cc:407] Connection header received
0812 02:36:51.183936 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:36:51.183940 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:36:51.184004 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:36:51.184063 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:36:51.184729 (+ 666us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40412: BlockingRecv error: recv got EOF from 127.0.0.1:40412 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260812 02:36:51.206321 7890 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:36:51.204137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40426 (local address 127.14.103.61:35237)
0812 02:36:51.204273 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0812 02:36:51.204278 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:36:51.204948 (+ 670us) server_negotiation.cc:407] Connection header received
0812 02:36:51.205201 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:36:51.205206 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:36:51.205288 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:36:51.205366 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:36:51.206193 (+ 827us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40426: BlockingRecv error: recv got EOF from 127.0.0.1:40426 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260812 02:36:51.246031 7890 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:36:51.241371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40438 (local address 127.14.103.61:35237)
0812 02:36:51.241521 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0812 02:36:51.241527 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 02:36:51.245024 (+ 3497us) server_negotiation.cc:407] Connection header received
0812 02:36:51.245209 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:36:51.245213 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:36:51.245275 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:36:51.245373 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:36:51.245863 (+ 490us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40438: BlockingRecv error: recv got EOF from 127.0.0.1:40438 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260812 02:36:51.313809 7890 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:36:51.312434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40448 (local address 127.14.103.61:35237)
0812 02:36:51.312617 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0812 02:36:51.312622 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:36:51.312825 (+ 203us) server_negotiation.cc:407] Connection header received
0812 02:36:51.313019 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:36:51.313023 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:36:51.313081 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:36:51.313184 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:36:51.313673 (+ 489us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40448: BlockingRecv error: recv got EOF from 127.0.0.1:40448 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
W20260812 02:36:51.445564 7890 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:36:51.444213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40454 (local address 127.14.103.61:35237)
0812 02:36:51.444379 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0812 02:36:51.444384 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:36:51.444593 (+ 209us) server_negotiation.cc:407] Connection header received
0812 02:36:51.444766 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:36:51.444770 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:36:51.444820 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:36:51.444918 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:36:51.445439 (+ 521us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40454: BlockingRecv error: recv got EOF from 127.0.0.1:40454 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
W20260812 02:36:51.705513 7890 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:36:51.703995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40462 (local address 127.14.103.61:35237)
0812 02:36:51.704275 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0812 02:36:51.704279 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:36:51.704460 (+ 181us) server_negotiation.cc:407] Connection header received
0812 02:36:51.704630 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:36:51.704633 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:36:51.704686 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:36:51.704766 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:36:51.705336 (+ 570us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40462: BlockingRecv error: recv got EOF from 127.0.0.1:40462 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
I20260812 02:36:52.009267 8002 catalog_manager.cc:1259] Loaded cluster ID: 4b974d3fbed541d6b9f59e0df6d3002f
I20260812 02:36:52.009404 8002 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d52e652d44204f9aad62c52c10a3f0fd: loading cluster ID for follower catalog manager: success
I20260812 02:36:52.011289 8002 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d52e652d44204f9aad62c52c10a3f0fd: acquiring CA information for follower catalog manager: success
I20260812 02:36:52.012761 8002 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d52e652d44204f9aad62c52c10a3f0fd: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 02:36:52.019765 8029 catalog_manager.cc:1259] Loaded cluster ID: 4b974d3fbed541d6b9f59e0df6d3002f
I20260812 02:36:52.019860 8029 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 510846af7c5947e58d7014c23dd0c05a: loading cluster ID for follower catalog manager: success
I20260812 02:36:52.021641 8029 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 510846af7c5947e58d7014c23dd0c05a: acquiring CA information for follower catalog manager: success
I20260812 02:36:52.022897 8029 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 510846af7c5947e58d7014c23dd0c05a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 02:36:52.224401 8118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:52.220240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40476 (local address 127.14.103.61:35237)
0812 02:36:52.220476 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0812 02:36:52.220479 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 02:36:52.220495 (+ 16us) server_negotiation.cc:407] Connection header received
0812 02:36:52.220549 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:36:52.220551 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:36:52.220672 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:36:52.220772 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 02:36:52.221056 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:52.221608 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:52.222571 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:52.222852 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:52.223004 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:36:52.223441 (+ 437us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 02:36:52.223932 (+ 491us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 02:36:52.224034 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0812 02:36:52.224193 (+ 159us) server_negotiation.cc:299] Negotiation successful
0812 02:36:52.224259 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":94,"threads_started":1}
I20260812 02:36:52.232126 8125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:52.228100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43904 (local address 127.14.103.60:45435)
0812 02:36:52.228348 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0812 02:36:52.228351 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 02:36:52.228366 (+ 15us) server_negotiation.cc:407] Connection header received
0812 02:36:52.228462 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:36:52.228464 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:36:52.228585 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:36:52.228690 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 02:36:52.228991 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:52.229526 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:52.230401 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:52.230645 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:52.230737 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:36:52.231140 (+ 403us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 02:36:52.231601 (+ 461us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 02:36:52.231703 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0812 02:36:52.231840 (+ 137us) server_negotiation.cc:299] Negotiation successful
0812 02:36:52.231898 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":101,"threads_started":1}
I20260812 02:36:52.239601 8118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:52.235448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40488 (local address 127.14.103.61:35237)
0812 02:36:52.235815 (+ 367us) server_negotiation.cc:206] Beginning negotiation
0812 02:36:52.235821 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 02:36:52.236085 (+ 264us) server_negotiation.cc:407] Connection header received
0812 02:36:52.236184 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:36:52.236190 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:36:52.236266 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:36:52.236352 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 02:36:52.236603 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:52.237439 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:52.238457 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:52.238696 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:52.238809 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:36:52.239091 (+ 282us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 02:36:52.239175 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 02:36:52.239259 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0812 02:36:52.239387 (+ 128us) server_negotiation.cc:299] Negotiation successful
0812 02:36:52.239459 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":202}
I20260812 02:36:52.240573 8125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:52.235467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43908 (local address 127.14.103.60:45435)
0812 02:36:52.235592 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0812 02:36:52.235596 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:36:52.236086 (+ 490us) server_negotiation.cc:407] Connection header received
0812 02:36:52.236285 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:36:52.236288 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:36:52.236345 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:36:52.236422 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 02:36:52.236734 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:52.237511 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:52.238862 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:52.239241 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:52.239327 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:36:52.239888 (+ 561us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 02:36:52.239966 (+ 78us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 02:36:52.240010 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0812 02:36:52.240361 (+ 351us) server_negotiation.cc:299] Negotiation successful
0812 02:36:52.240418 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260812 02:36:52.240811 8132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:52.235451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39952 (local address 127.14.103.62:39137)
0812 02:36:52.235944 (+ 493us) server_negotiation.cc:206] Beginning negotiation
0812 02:36:52.235949 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:36:52.235965 (+ 16us) server_negotiation.cc:407] Connection header received
0812 02:36:52.236074 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:36:52.236078 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:36:52.236281 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:36:52.236702 (+ 421us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 02:36:52.236850 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:52.237408 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:52.238862 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:52.239240 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:52.239344 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:36:52.239818 (+ 474us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 02:36:52.240196 (+ 378us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 02:36:52.240293 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0812 02:36:52.240584 (+ 291us) server_negotiation.cc:299] Negotiation successful
0812 02:36:52.240652 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":412,"thread_start_us":123,"threads_started":1}
I20260812 02:36:52.244524 14748 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbep4M4/build/debug/bin/kudu with pid 7722
I20260812 02:36:52.254884 14748 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbep4M4/build/debug/bin/kudu with pid 7807
I20260812 02:36:52.264017 14748 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbep4M4/build/debug/bin/kudu with pid 7891
2026-08-12T02:36:52Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3253 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786501898713699-14748-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 02:36:52 dist-test-slave-2w3w krb5kdc[8137](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 02:36:52 dist-test-slave-2w3w krb5kdc[8137](info): set up 2 sockets
Aug 12 02:36:52 dist-test-slave-2w3w krb5kdc[8137](info): commencing operation
krb5kdc: starting...
W20260812 02:36:53.462208 14748 mini_kdc.cc:121] Time spent starting KDC: real 1.172s user 0.006s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786501898713699-14748-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786501898713699-14748-0/krb5kdc/test-user.keytab.
Aug 12 02:36:53 dist-test-slave-2w3w krb5kdc[8137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502213, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T02:36:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T02:36:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.103.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.103.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.103.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.62.keytab.
Entry for principal kudu/127.14.103.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.62.keytab.
I20260812 02:36:53.603020 14748 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbep4M4/build/debug/bin/kudu
/tmp/dist-test-taskbep4M4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786501898713699-14748-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786501898713699-14748-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786501898713699-14748-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786501898713699-14748-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.103.62:38107
--webserver_interface=127.14.103.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.62.keytab
--principal=kudu/127.14.103.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.103.20:42421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.103.62:38107,127.14.103.61:36977,127.14.103.60:40291
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786501898713699-14748-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786501898713699-14748-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786501898713699-14748-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:36:53.718365 8152 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260812 02:36:53.718678 8152 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:36:53.718704 8152 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:36:53.718806 8152 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:36:53.722954 8152 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:36:53.723055 8152 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:36:53.723074 8152 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:36:53.723093 8152 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:36:53.723107 8152 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 02:36:53.728700 8152 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.103.20:42421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786501898713699-14748-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786501898713699-14748-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.103.62:38107,127.14.103.61:36977,127.14.103.60:40291
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.62.keytab
--principal=kudu/127.14.103.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.103.62:38107
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786501898713699-14748-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.103.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8152
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786501898713699-14748-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:36:53.729888 8152 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:36:53.731024 8152 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:36:53.739884 8161 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:36:53.739889 8157 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:36:53.739889 8158 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:36:53.740314 8152 server_base.cc:1034] running on GCE node
I20260812 02:36:53.740909 8152 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:36:53.742055 8152 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:36:53.743224 8152 hybrid_clock.cc:648] HybridClock initialized: now 1786502213743192 us; error 44 us; skew 500 ppm
Aug 12 02:36:53 dist-test-slave-2w3w krb5kdc[8137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502213, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:36:53.746132 8152 init.cc:377] Logged in from keytab as kudu/127.14.103.62@KRBTEST.COM (short username kudu)
I20260812 02:36:53.747159 8152 webserver.cc:460] Webserver started at http://127.14.103.62:35649/ using document root <none> and password file <none>
I20260812 02:36:53.747699 8152 fs_manager.cc:359] Metadata directory not provided
I20260812 02:36:53.747761 8152 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:36:53.747931 8152 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:36:53.749975 8152 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786501898713699-14748-0/minicluster-data/master-0/data/instance:
uuid: "0500951028ea449cb391b558b03cc234"
format_stamp: "Formatted at 2026-08-12 02:36:53 on dist-test-slave-2w3w"
server_key: "464d829d2a4964aa5f97ee581d4234d9"
server_key_iv: "ce4cef2d29a24d2b74a2658c6ad76891"
server_key_version: "encryptionkey@0"
I20260812 02:36:53.750520 8152 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786501898713699-14748-0/minicluster-data/master-0/wal/instance:
uuid: "0500951028ea449cb391b558b03cc234"
format_stamp: "Formatted at 2026-08-12 02:36:53 on dist-test-slave-2w3w"
server_key: "464d829d2a4964aa5f97ee581d4234d9"
server_key_iv: "ce4cef2d29a24d2b74a2658c6ad76891"
server_key_version: "encryptionkey@0"
I20260812 02:36:53.754421 8152 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260812 02:36:53.757000 8169 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:36:53.758420 8152 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260812 02:36:53.758535 8152 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786501898713699-14748-0/minicluster-data/master-0/data,/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786501898713699-14748-0/minicluster-data/master-0/wal
uuid: "0500951028ea449cb391b558b03cc234"
format_stamp: "Formatted at 2026-08-12 02:36:53 on dist-test-slave-2w3w"
server_key: "464d829d2a4964aa5f97ee581d4234d9"
server_key_iv: "ce4cef2d29a24d2b74a2658c6ad76891"
server_key_version: "encryptionkey@0"
I20260812 02:36:53.758635 8152 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786501898713699-14748-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786501898713699-14748-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786501898713699-14748-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:36:53.773190 8152 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:36:53.776616 8152 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:36:53.776798 8152 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:36:53.785000 8152 rpc_server.cc:307] RPC server started. Bound to: 127.14.103.62:38107
I20260812 02:36:53.785338 8228 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.103.62:38107 every 8 connection(s)
I20260812 02:36:53.786103 8152 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786501898713699-14748-0/minicluster-data/master-0/data/info.pb
I20260812 02:36:53.789037 8229 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:36:53.790951 14748 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbep4M4/build/debug/bin/kudu as pid 8152
I20260812 02:36:53.791131 14748 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786501898713699-14748-0/minicluster-data/master-0/wal/instance
I20260812 02:36:53.791615 14748 external_mini_cluster.cc:1428] Setting key 6c67a8b700634e8075bdc47237681ef3
I20260812 02:36:53.794677 8229 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.103.62" port: 38107 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.14.103.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.103.61@KRBTEST.COM" created.
Aug 12 02:36:53 dist-test-slave-2w3w krb5kdc[8137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502213, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.62@KRBTEST.COM for kudu/127.14.103.62@KRBTEST.COM
I20260812 02:36:53.812162 8233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:53.797362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:38107 (local address 127.0.0.1:43714)
0812 02:36:53.798032 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0812 02:36:53.798070 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0812 02:36:53.799037 (+ 967us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:36:53.799817 (+ 780us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:36:53.799822 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:36:53.800287 (+ 465us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:36:53.800534 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:36:53.800548 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:36:53.801852 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:36:53.801873 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:36:53.802823 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:36:53.802836 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:36:53.803624 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:36:53.803629 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:36:53.803752 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:36:53.804479 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:36:53.804508 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:36:53.806565 (+ 2057us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:36:53.808831 (+ 2266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:36:53.808841 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:36:53.808853 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:36:53.809129 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:36:53.809478 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:36:53.809483 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:36:53.809485 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:36:53.809611 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:36:53.810262 (+ 651us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:36:53.810269 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:36:53.810423 (+ 154us) client_negotiation.cc:769] Sending connection context
0812 02:36:53.810609 (+ 186us) client_negotiation.cc:240] Negotiation successful
0812 02:36:53.811060 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":508,"mutex_wait_us":135,"thread_start_us":168,"threads_started":1}
I20260812 02:36:53.812162 8232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:53.797361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43714 (local address 127.14.103.62:38107)
0812 02:36:53.797862 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0812 02:36:53.797883 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0812 02:36:53.798191 (+ 308us) server_negotiation.cc:407] Connection header received
0812 02:36:53.799368 (+ 1177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:36:53.799374 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:36:53.799600 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:36:53.799823 (+ 223us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:36:53.800688 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:53.801658 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:53.803005 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:53.803442 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:53.803641 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:36:53.806751 (+ 3110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:36:53.806758 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:36:53.806770 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:36:53.806793 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:36:53.808680 (+ 1887us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:36:53.809251 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:36:53.809262 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:36:53.809264 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:36:53.809375 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:36:53.809708 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:36:53.809714 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:36:53.809716 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:36:53.810137 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:36:53.810285 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0812 02:36:53.810672 (+ 387us) server_negotiation.cc:299] Negotiation successful
0812 02:36:53.811011 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":27,"server-negotiator.queue_time_us":327,"thread_start_us":175,"threads_started":1}
I20260812 02:36:53.815378 8229 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.103.61" port: 36977 } has no permanent_uuid. Determining permanent_uuid...
I20260812 02:36:53.816473 8233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:53.816049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.61:36977 (local address 127.0.0.1:57984)
0812 02:36:53.816305 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0812 02:36:53.816394 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.61:36977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155}
W20260812 02:36:53.816816 8174 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.103.61:36977: connect: Connection refused (error 111)
I20260812 02:36:53.817842 8233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:53.817595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.61:36977 (local address 127.0.0.1:57998)
0812 02:36:53.817720 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0812 02:36:53.817774 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.61:36977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260812 02:36:53.818130 8229 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.103.61:36977: Network error: Client connection negotiation failed: client connection to 127.14.103.61:36977: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.103.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.61.keytab.
Entry for principal kudu/127.14.103.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.61.keytab.
I20260812 02:36:53.827950 14748 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbep4M4/build/debug/bin/kudu
/tmp/dist-test-taskbep4M4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786501898713699-14748-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786501898713699-14748-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786501898713699-14748-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786501898713699-14748-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.103.61:36977
--webserver_interface=127.14.103.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.61.keytab
--principal=kudu/127.14.103.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.103.20:42421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.103.62:38107,127.14.103.61:36977,127.14.103.60:40291
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786501898713699-14748-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786501898713699-14748-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786501898713699-14748-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 02:36:53.867445 8229 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.103.61" port: 36977 } attempt: 1
I20260812 02:36:53.868407 8233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:53.868041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.61:36977 (local address 127.0.0.1:58010)
0812 02:36:53.868222 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0812 02:36:53.868308 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.61:36977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260812 02:36:53.869274 8233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:53.869010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.61:36977 (local address 127.0.0.1:58014)
0812 02:36:53.869131 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0812 02:36:53.869190 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.61:36977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260812 02:36:53.869432 8229 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.103.61:36977: Network error: Client connection negotiation failed: client connection to 127.14.103.61:36977: connect: Connection refused (error 111)
I20260812 02:36:53.937674 8229 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.103.61" port: 36977 } attempt: 2
I20260812 02:36:53.938664 8233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:53.938305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.61:36977 (local address 127.0.0.1:58024)
0812 02:36:53.938493 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0812 02:36:53.938569 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.61:36977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260812 02:36:53.939493 8233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:53.939238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.61:36977 (local address 127.0.0.1:58026)
0812 02:36:53.939367 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0812 02:36:53.939424 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.61:36977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260812 02:36:53.939661 8229 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.103.61:36977: Network error: Client connection negotiation failed: client connection to 127.14.103.61:36977: connect: Connection refused (error 111)
W20260812 02:36:53.948040 8236 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260812 02:36:53.948443 8236 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:36:53.948518 8236 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:36:53.948601 8236 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:36:53.953037 8236 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:36:53.953166 8236 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:36:53.953202 8236 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:36:53.953246 8236 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:36:53.953305 8236 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 02:36:53.959764 8236 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.103.20:42421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786501898713699-14748-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786501898713699-14748-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.14.103.62:38107,127.14.103.61:36977,127.14.103.60:40291
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.61.keytab
--principal=kudu/127.14.103.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.103.61:36977
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786501898713699-14748-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.14.103.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8236
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786501898713699-14748-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:36:53.961037 8236 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:36:53.962224 8236 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:36:53.973167 8245 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:36:53.973101 8243 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:36:53.973101 8247 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:36:53.973403 8236 server_base.cc:1034] running on GCE node
I20260812 02:36:53.974205 8236 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:36:53.975530 8236 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:36:53.976758 8236 hybrid_clock.cc:648] HybridClock initialized: now 1786502213976733 us; error 40 us; skew 500 ppm
Aug 12 02:36:53 dist-test-slave-2w3w krb5kdc[8137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502213, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:36:53.980100 8236 init.cc:377] Logged in from keytab as kudu/127.14.103.61@KRBTEST.COM (short username kudu)
I20260812 02:36:53.981302 8236 webserver.cc:460] Webserver started at http://127.14.103.61:34997/ using document root <none> and password file <none>
I20260812 02:36:53.981992 8236 fs_manager.cc:359] Metadata directory not provided
I20260812 02:36:53.982100 8236 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:36:53.982363 8236 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:36:53.984917 8236 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786501898713699-14748-0/minicluster-data/master-1/data/instance:
uuid: "9aa8b2189f974391bcd27b61ea5dea46"
format_stamp: "Formatted at 2026-08-12 02:36:53 on dist-test-slave-2w3w"
server_key: "dafce33c990f622ad74c05bd5911f08a"
server_key_iv: "57193812ed1846f3fa2c47c6d068d90b"
server_key_version: "encryptionkey@0"
I20260812 02:36:53.985581 8236 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786501898713699-14748-0/minicluster-data/master-1/wal/instance:
uuid: "9aa8b2189f974391bcd27b61ea5dea46"
format_stamp: "Formatted at 2026-08-12 02:36:53 on dist-test-slave-2w3w"
server_key: "dafce33c990f622ad74c05bd5911f08a"
server_key_iv: "57193812ed1846f3fa2c47c6d068d90b"
server_key_version: "encryptionkey@0"
I20260812 02:36:53.989761 8236 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260812 02:36:53.992635 8254 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:36:53.994180 8236 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260812 02:36:53.994355 8236 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786501898713699-14748-0/minicluster-data/master-1/data,/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786501898713699-14748-0/minicluster-data/master-1/wal
uuid: "9aa8b2189f974391bcd27b61ea5dea46"
format_stamp: "Formatted at 2026-08-12 02:36:53 on dist-test-slave-2w3w"
server_key: "dafce33c990f622ad74c05bd5911f08a"
server_key_iv: "57193812ed1846f3fa2c47c6d068d90b"
server_key_version: "encryptionkey@0"
I20260812 02:36:53.994503 8236 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786501898713699-14748-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786501898713699-14748-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786501898713699-14748-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:36:54.011875 8236 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:36:54.016077 8236 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:36:54.016340 8236 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:36:54.024657 8236 rpc_server.cc:307] RPC server started. Bound to: 127.14.103.61:36977
I20260812 02:36:54.024969 8311 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.103.61:36977 every 8 connection(s)
I20260812 02:36:54.025844 8236 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786501898713699-14748-0/minicluster-data/master-1/data/info.pb
I20260812 02:36:54.026607 14748 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbep4M4/build/debug/bin/kudu as pid 8236
I20260812 02:36:54.026754 14748 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786501898713699-14748-0/minicluster-data/master-1/wal/instance
I20260812 02:36:54.027158 14748 external_mini_cluster.cc:1428] Setting key f0d6c916b3254800fd662f97733bdaa0
I20260812 02:36:54.029671 8312 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:36:54.030848 8229 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.103.61" port: 36977 } attempt: 3
I20260812 02:36:54.036010 8312 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.103.62" port: 38107 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.14.103.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.103.60@KRBTEST.COM" created.
Aug 12 02:36:54 dist-test-slave-2w3w krb5kdc[8137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502213, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.62@KRBTEST.COM for kudu/127.14.103.61@KRBTEST.COM
Aug 12 02:36:54 dist-test-slave-2w3w krb5kdc[8137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502213, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.61@KRBTEST.COM for kudu/127.14.103.62@KRBTEST.COM
I20260812 02:36:54.047796 8233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:54.031486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.61:36977 (local address 127.0.0.1:58032)
0812 02:36:54.031616 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0812 02:36:54.031634 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 02:36:54.031736 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:36:54.035296 (+ 3560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:36:54.035302 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:36:54.035527 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:36:54.035682 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:36:54.035691 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:36:54.037660 (+ 1969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:36:54.037667 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:36:54.038499 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:36:54.038512 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:36:54.039568 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:36:54.039571 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:36:54.039665 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:36:54.040337 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:36:54.040352 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:36:54.042629 (+ 2277us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:36:54.045216 (+ 2587us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:36:54.045223 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:36:54.045228 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:36:54.045569 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:36:54.046002 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:36:54.046008 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:36:54.046013 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:36:54.046119 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:36:54.047031 (+ 912us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:36:54.047037 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:36:54.047181 (+ 144us) client_negotiation.cc:769] Sending connection context
0812 02:36:54.047317 (+ 136us) client_negotiation.cc:240] Negotiation successful
0812 02:36:54.047505 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260812 02:36:54.049098 8232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:54.038165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43726 (local address 127.14.103.62:38107)
0812 02:36:54.038282 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0812 02:36:54.038287 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:36:54.038553 (+ 266us) server_negotiation.cc:407] Connection header received
0812 02:36:54.038731 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:36:54.038734 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:36:54.038796 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:36:54.038882 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:36:54.039823 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:54.040470 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:54.041757 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:54.042168 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:54.042303 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:36:54.045320 (+ 3017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:36:54.045322 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:36:54.045324 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:36:54.045349 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:36:54.047062 (+ 1713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:36:54.047599 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:36:54.047603 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:36:54.047607 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:36:54.047717 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:36:54.048085 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:36:54.048091 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:36:54.048095 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:36:54.048318 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:36:54.048389 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0812 02:36:54.048813 (+ 424us) server_negotiation.cc:299] Negotiation successful
0812 02:36:54.048943 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260812 02:36:54.049559 8317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:54.032328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58032 (local address 127.14.103.61:36977)
0812 02:36:54.032969 (+ 641us) server_negotiation.cc:206] Beginning negotiation
0812 02:36:54.032990 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0812 02:36:54.033042 (+ 52us) server_negotiation.cc:407] Connection header received
0812 02:36:54.034247 (+ 1205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:36:54.034291 (+ 44us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:36:54.034841 (+ 550us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:36:54.035342 (+ 501us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:36:54.035811 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:54.037501 (+ 1690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:54.038655 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:54.039408 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:54.039582 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:36:54.042816 (+ 3234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:36:54.042827 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:36:54.042843 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:36:54.042883 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:36:54.045092 (+ 2209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:36:54.045709 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:36:54.045718 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:36:54.045728 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:36:54.045877 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:36:54.046236 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:36:54.046241 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:36:54.046246 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:36:54.046856 (+ 610us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:36:54.047724 (+ 868us) server_negotiation.cc:1035] Waiting for connection context
0812 02:36:54.047986 (+ 262us) server_negotiation.cc:299] Negotiation successful
0812 02:36:54.048355 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":443,"spinlock_wait_cycles":19328,"thread_start_us":155,"threads_started":1}
I20260812 02:36:54.049566 8318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:54.037852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:38107 (local address 127.0.0.1:43726)
0812 02:36:54.038427 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0812 02:36:54.038467 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0812 02:36:54.038622 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:36:54.038954 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:36:54.038983 (+ 29us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:36:54.039479 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:36:54.039642 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:36:54.039649 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:36:54.040717 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:36:54.040720 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:36:54.041558 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:36:54.041575 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:36:54.042365 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:36:54.042371 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:36:54.042520 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:36:54.043245 (+ 725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:36:54.043265 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:36:54.045185 (+ 1920us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:36:54.047180 (+ 1995us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:36:54.047191 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:36:54.047197 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:36:54.047445 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:36:54.047863 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:36:54.047866 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:36:54.047868 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:36:54.047967 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:36:54.048477 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:36:54.048482 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:36:54.048580 (+ 98us) client_negotiation.cc:769] Sending connection context
0812 02:36:54.048724 (+ 144us) client_negotiation.cc:240] Negotiation successful
0812 02:36:54.048954 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":176,"threads_started":1}
I20260812 02:36:54.050859 8312 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.103.61" port: 36977 } has no permanent_uuid. Determining permanent_uuid...
I20260812 02:36:54.052103 8229 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.103.60" port: 40291 } has no permanent_uuid. Determining permanent_uuid...
I20260812 02:36:54.052877 8233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:54.052595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.60:40291 (local address 127.0.0.1:58736)
0812 02:36:54.052767 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0812 02:36:54.052815 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.60:40291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
I20260812 02:36:54.053647 8233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:54.053321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.60:40291 (local address 127.0.0.1:58750)
0812 02:36:54.053469 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0812 02:36:54.053527 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.60:40291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
W20260812 02:36:54.053817 8229 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.103.60:40291: Network error: Client connection negotiation failed: client connection to 127.14.103.60:40291: connect: Connection refused (error 111)
Aug 12 02:36:54 dist-test-slave-2w3w krb5kdc[8137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502213, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.61@KRBTEST.COM for kudu/127.14.103.61@KRBTEST.COM
I20260812 02:36:54.061898 8318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:54.051459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.61:36977 (local address 127.0.0.1:58034)
0812 02:36:54.051561 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0812 02:36:54.051575 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 02:36:54.051654 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:36:54.052231 (+ 577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:36:54.052234 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:36:54.052444 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:36:54.052575 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:36:54.052583 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:36:54.053480 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:36:54.053486 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:36:54.054209 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:36:54.054219 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:36:54.054701 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:36:54.054704 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:36:54.054815 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:36:54.055377 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:36:54.055393 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:36:54.056782 (+ 1389us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:36:54.059911 (+ 3129us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:36:54.059914 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:36:54.059916 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:36:54.060160 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:36:54.060631 (+ 471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:36:54.060634 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:36:54.060635 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:36:54.060728 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:36:54.061333 (+ 605us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:36:54.061336 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:36:54.061428 (+ 92us) client_negotiation.cc:769] Sending connection context
0812 02:36:54.061563 (+ 135us) client_negotiation.cc:240] Negotiation successful
0812 02:36:54.061763 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260812 02:36:54.062228 8320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:54.051630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58034 (local address 127.14.103.61:36977)
0812 02:36:54.052013 (+ 383us) server_negotiation.cc:206] Beginning negotiation
0812 02:36:54.052022 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 02:36:54.052035 (+ 13us) server_negotiation.cc:407] Connection header received
0812 02:36:54.052073 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:36:54.052075 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:36:54.052118 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:36:54.052222 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:36:54.052709 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:54.053366 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:54.054332 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:54.054590 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:54.054702 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:36:54.057607 (+ 2905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:36:54.057616 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:36:54.057622 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:36:54.057651 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:36:54.059716 (+ 2065us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:36:54.060393 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:36:54.060397 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:36:54.060398 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:36:54.060488 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:36:54.060862 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:36:54.060865 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:36:54.060866 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:36:54.061181 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:36:54.061285 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0812 02:36:54.061623 (+ 338us) server_negotiation.cc:299] Negotiation successful
0812 02:36:54.061965 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":131,"server-negotiator.queue_time_us":308,"thread_start_us":132,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.103.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.60.keytab.
Entry for principal kudu/127.14.103.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.60.keytab.
I20260812 02:36:54.063077 8312 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.103.60" port: 40291 } has no permanent_uuid. Determining permanent_uuid...
I20260812 02:36:54.063903 8318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:54.063569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.60:40291 (local address 127.0.0.1:58766)
0812 02:36:54.063722 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0812 02:36:54.063831 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.60:40291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260812 02:36:54.063898 14748 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbep4M4/build/debug/bin/kudu
/tmp/dist-test-taskbep4M4/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786501898713699-14748-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786501898713699-14748-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786501898713699-14748-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786501898713699-14748-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.103.60:40291
--webserver_interface=127.14.103.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.60.keytab
--principal=kudu/127.14.103.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.103.20:42421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.103.62:38107,127.14.103.61:36977,127.14.103.60:40291
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786501898713699-14748-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786501898713699-14748-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786501898713699-14748-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:36:54.064191 8257 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.103.60:40291: connect: Connection refused (error 111)
I20260812 02:36:54.065222 8318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:54.064945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.60:40291 (local address 127.0.0.1:58782)
0812 02:36:54.065077 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0812 02:36:54.065140 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.60:40291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260812 02:36:54.065421 8312 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.103.60:40291: Network error: Client connection negotiation failed: client connection to 127.14.103.60:40291: connect: Connection refused (error 111)
I20260812 02:36:54.085021 8229 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.103.60" port: 40291 } attempt: 1
I20260812 02:36:54.086004 8233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:54.085655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.60:40291 (local address 127.0.0.1:58784)
0812 02:36:54.085839 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0812 02:36:54.085909 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.60:40291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260812 02:36:54.086902 8233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:54.086642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.60:40291 (local address 127.0.0.1:58794)
0812 02:36:54.086783 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0812 02:36:54.086834 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.60:40291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260812 02:36:54.087085 8229 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.103.60:40291: Network error: Client connection negotiation failed: client connection to 127.14.103.60:40291: connect: Connection refused (error 111)
I20260812 02:36:54.114785 8312 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.103.60" port: 40291 } attempt: 1
I20260812 02:36:54.115839 8318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:54.115507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.60:40291 (local address 127.0.0.1:58804)
0812 02:36:54.115688 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0812 02:36:54.115758 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.60:40291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260812 02:36:54.116699 8318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:54.116450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.60:40291 (local address 127.0.0.1:58806)
0812 02:36:54.116585 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0812 02:36:54.116636 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.60:40291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260812 02:36:54.116894 8312 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.103.60:40291: Network error: Client connection negotiation failed: client connection to 127.14.103.60:40291: connect: Connection refused (error 111)
I20260812 02:36:54.162384 8229 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.103.60" port: 40291 } attempt: 2
I20260812 02:36:54.163414 8233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:54.163019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.60:40291 (local address 127.0.0.1:58820)
0812 02:36:54.163243 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0812 02:36:54.163327 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.60:40291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120}
I20260812 02:36:54.164175 8233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:54.163946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.60:40291 (local address 127.0.0.1:58830)
0812 02:36:54.164062 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0812 02:36:54.164118 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.60:40291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260812 02:36:54.164325 8229 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.103.60:40291: Network error: Client connection negotiation failed: client connection to 127.14.103.60:40291: connect: Connection refused (error 111)
I20260812 02:36:54.185078 8312 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.103.60" port: 40291 } attempt: 2
W20260812 02:36:54.184955 8323 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260812 02:36:54.185325 8323 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:36:54.185403 8323 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:36:54.185488 8323 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260812 02:36:54.186067 8318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:54.185735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.60:40291 (local address 127.0.0.1:58838)
0812 02:36:54.185923 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0812 02:36:54.185988 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.60:40291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260812 02:36:54.186759 8318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:54.186552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.60:40291 (local address 127.0.0.1:58852)
0812 02:36:54.186661 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0812 02:36:54.186712 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.103.60:40291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260812 02:36:54.186908 8312 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.103.60:40291: Network error: Client connection negotiation failed: client connection to 127.14.103.60:40291: connect: Connection refused (error 111)
W20260812 02:36:54.189911 8323 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:36:54.190001 8323 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:36:54.190039 8323 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:36:54.190114 8323 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:36:54.190145 8323 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 02:36:54.195978 8323 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.103.20:42421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786501898713699-14748-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786501898713699-14748-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.14.103.62:38107,127.14.103.61:36977,127.14.103.60:40291
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786501898713699-14748-0/krb5kdc/kudu_127.14.103.60.keytab
--principal=kudu/127.14.103.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.103.60:40291
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786501898713699-14748-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.14.103.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8323
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786501898713699-14748-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 01:57:48 UTC on ccycloud.quasar-pvzurd.root.comops.site
build id 4581823
I20260812 02:36:54.197098 8323 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:36:54.198166 8323 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:36:54.207136 8328 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:36:54.207195 8329 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:36:54.207311 8323 server_base.cc:1034] running on GCE node
W20260812 02:36:54.207136 8331 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:36:54.208149 8323 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:36:54.209299 8323 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:36:54.210516 8323 hybrid_clock.cc:648] HybridClock initialized: now 1786502214210489 us; error 45 us; skew 500 ppm
Aug 12 02:36:54 dist-test-slave-2w3w krb5kdc[8137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502214, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:36:54.213565 8323 init.cc:377] Logged in from keytab as kudu/127.14.103.60@KRBTEST.COM (short username kudu)
I20260812 02:36:54.214524 8323 webserver.cc:460] Webserver started at http://127.14.103.60:45479/ using document root <none> and password file <none>
I20260812 02:36:54.215194 8323 fs_manager.cc:359] Metadata directory not provided
I20260812 02:36:54.215296 8323 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:36:54.215538 8323 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:36:54.217674 8323 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786501898713699-14748-0/minicluster-data/master-2/data/instance:
uuid: "f9ba6a023c574841b6a4960ea035990b"
format_stamp: "Formatted at 2026-08-12 02:36:54 on dist-test-slave-2w3w"
server_key: "07de435a106886c84b779aed47e04174"
server_key_iv: "dbc0ad2023626238b53a4b210ab3fd5c"
server_key_version: "encryptionkey@0"
I20260812 02:36:54.218294 8323 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786501898713699-14748-0/minicluster-data/master-2/wal/instance:
uuid: "f9ba6a023c574841b6a4960ea035990b"
format_stamp: "Formatted at 2026-08-12 02:36:54 on dist-test-slave-2w3w"
server_key: "07de435a106886c84b779aed47e04174"
server_key_iv: "dbc0ad2023626238b53a4b210ab3fd5c"
server_key_version: "encryptionkey@0"
I20260812 02:36:54.222477 8323 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260812 02:36:54.225216 8339 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:36:54.226593 8323 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260812 02:36:54.226763 8323 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786501898713699-14748-0/minicluster-data/master-2/data,/tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786501898713699-14748-0/minicluster-data/master-2/wal
uuid: "f9ba6a023c574841b6a4960ea035990b"
format_stamp: "Formatted at 2026-08-12 02:36:54 on dist-test-slave-2w3w"
server_key: "07de435a106886c84b779aed47e04174"
server_key_iv: "dbc0ad2023626238b53a4b210ab3fd5c"
server_key_version: "encryptionkey@0"
I20260812 02:36:54.226900 8323 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786501898713699-14748-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786501898713699-14748-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786501898713699-14748-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:36:54.249882 8323 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:36:54.253295 8323 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:36:54.253515 8323 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:36:54.260820 8323 rpc_server.cc:307] RPC server started. Bound to: 127.14.103.60:40291
I20260812 02:36:54.261143 8398 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.103.60:40291 every 8 connection(s)
I20260812 02:36:54.261859 8323 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786501898713699-14748-0/minicluster-data/master-2/data/info.pb
I20260812 02:36:54.262514 14748 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbep4M4/build/debug/bin/kudu as pid 8323
I20260812 02:36:54.262660 14748 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbep4M4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786501898713699-14748-0/minicluster-data/master-2/wal/instance
I20260812 02:36:54.263087 14748 external_mini_cluster.cc:1428] Setting key 2df469703a42ace2615db0c76dca6b5e
I20260812 02:36:54.263482 8229 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.103.60" port: 40291 } attempt: 3
I20260812 02:36:54.266077 8399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:36:54.270853 8399 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.103.62" port: 38107 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 02:36:54 dist-test-slave-2w3w krb5kdc[8137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502213, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.62@KRBTEST.COM for kudu/127.14.103.60@KRBTEST.COM
Aug 12 02:36:54 dist-test-slave-2w3w krb5kdc[8137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502213, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.103.60@KRBTEST.COM
I20260812 02:36:54.278053 8233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:54.263896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.60:40291 (local address 127.0.0.1:58870)
0812 02:36:54.264032 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0812 02:36:54.264052 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 02:36:54.264149 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:36:54.266555 (+ 2406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:36:54.266559 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:36:54.266782 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:36:54.266915 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:36:54.266924 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:36:54.268666 (+ 1742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:36:54.268670 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:36:54.269509 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:36:54.269522 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:36:54.270372 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:36:54.270376 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:36:54.270472 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:36:54.271133 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:36:54.271150 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:36:54.272840 (+ 1690us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:36:54.275704 (+ 2864us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:36:54.275709 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:36:54.275711 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:36:54.275979 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:36:54.276419 (+ 440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:36:54.276422 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:36:54.276424 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:36:54.276529 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:36:54.277388 (+ 859us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:36:54.277391 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:36:54.277600 (+ 209us) client_negotiation.cc:769] Sending connection context
0812 02:36:54.277706 (+ 106us) client_negotiation.cc:240] Negotiation successful
0812 02:36:54.277856 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55,"mutex_wait_us":115}
I20260812 02:36:54.278124 8312 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.103.60" port: 40291 } attempt: 3
I20260812 02:36:54.279378 8402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:54.264600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58868 (local address 127.14.103.60:40291)
0812 02:36:54.265033 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0812 02:36:54.265053 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0812 02:36:54.265097 (+ 44us) server_negotiation.cc:407] Connection header received
0812 02:36:54.265787 (+ 690us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:36:54.265803 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:36:54.266207 (+ 404us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:36:54.266576 (+ 369us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:36:54.267372 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:54.268424 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:54.269682 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:54.270185 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:54.270385 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:36:54.273950 (+ 3565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:36:54.273954 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:36:54.273956 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:36:54.273982 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:36:54.275832 (+ 1850us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:36:54.276324 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:36:54.276328 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:36:54.276330 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:36:54.276422 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:36:54.276721 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:36:54.276723 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:36:54.276725 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:36:54.277254 (+ 529us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:36:54.277441 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0812 02:36:54.277859 (+ 418us) server_negotiation.cc:299] Negotiation successful
0812 02:36:54.278233 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":48,"server-negotiator.queue_time_us":296,"spinlock_wait_cycles":12416,"thread_start_us":165,"threads_started":1}
I20260812 02:36:54.280515 8403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:54.265041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58870 (local address 127.14.103.60:40291)
0812 02:36:54.265331 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0812 02:36:54.265341 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 02:36:54.265356 (+ 15us) server_negotiation.cc:407] Connection header received
0812 02:36:54.265811 (+ 455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:36:54.265816 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:36:54.266378 (+ 562us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:36:54.267079 (+ 701us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:36:54.267208 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:54.268483 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:54.269652 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:54.270197 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:54.270714 (+ 517us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:36:54.273104 (+ 2390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:36:54.273112 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:36:54.273129 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:36:54.273165 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:36:54.275575 (+ 2410us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:36:54.276093 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:36:54.276099 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:36:54.276106 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:36:54.276250 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:36:54.276632 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:36:54.276637 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:36:54.276639 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:36:54.277254 (+ 615us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:36:54.277399 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0812 02:36:54.277859 (+ 460us) server_negotiation.cc:299] Negotiation successful
0812 02:36:54.279424 (+ 1565us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":476,"server-negotiator.queue_time_us":220,"spinlock_wait_cycles":8320,"thread_start_us":81,"threads_started":1}
Aug 12 02:36:54 dist-test-slave-2w3w krb5kdc[8137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502214, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.60@KRBTEST.COM for kudu/127.14.103.62@KRBTEST.COM
I20260812 02:36:54.284139 8229 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0500951028ea449cb391b558b03cc234: Bootstrap starting.
Aug 12 02:36:54 dist-test-slave-2w3w krb5kdc[8137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502213, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.61@KRBTEST.COM for kudu/127.14.103.60@KRBTEST.COM
I20260812 02:36:54.286767 8232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:54.272906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43730 (local address 127.14.103.62:38107)
0812 02:36:54.273045 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0812 02:36:54.273052 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 02:36:54.273475 (+ 423us) server_negotiation.cc:407] Connection header received
0812 02:36:54.273684 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:36:54.273688 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:36:54.273752 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:36:54.273852 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:36:54.274991 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:54.275844 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:54.277053 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:54.277301 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:54.277436 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:36:54.282498 (+ 5062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:36:54.282504 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:36:54.282508 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:36:54.282544 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:36:54.284531 (+ 1987us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:36:54.285243 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:36:54.285250 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:36:54.285265 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:36:54.285365 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:36:54.285685 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:36:54.285688 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:36:54.285690 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:36:54.285935 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:36:54.286007 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0812 02:36:54.286466 (+ 459us) server_negotiation.cc:299] Negotiation successful
0812 02:36:54.286592 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260812 02:36:54.287442 8229 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0500951028ea449cb391b558b03cc234: Neither blocks nor log segments found. Creating new log.
I20260812 02:36:54.288010 8404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:54.272884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.62:38107 (local address 127.0.0.1:43730)
0812 02:36:54.273347 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0812 02:36:54.273400 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0812 02:36:54.273563 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:36:54.273905 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:36:54.273911 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:36:54.274611 (+ 700us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:36:54.274811 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:36:54.274821 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:36:54.275971 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:36:54.275974 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:36:54.276938 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:36:54.276947 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:36:54.278554 (+ 1607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:36:54.278558 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:36:54.278807 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:36:54.279386 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:36:54.279404 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:36:54.282330 (+ 2926us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:36:54.284705 (+ 2375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:36:54.284716 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:36:54.284723 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:36:54.285123 (+ 400us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:36:54.285482 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:36:54.285484 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:36:54.285486 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:36:54.285577 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:36:54.286090 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:36:54.286096 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:36:54.286226 (+ 130us) client_negotiation.cc:769] Sending connection context
0812 02:36:54.287471 (+ 1245us) client_negotiation.cc:240] Negotiation successful
0812 02:36:54.287765 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":145,"threads_started":1}
I20260812 02:36:54.288569 8229 log.cc:824] T 00000000000000000000000000000000 P 0500951028ea449cb391b558b03cc234: Log is configured to *not* fsync() on all Append() calls
Aug 12 02:36:54 dist-test-slave-2w3w krb5kdc[8137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502213, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.103.62@KRBTEST.COM
I20260812 02:36:54.289532 8399 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.103.61" port: 36977 } has no permanent_uuid. Determining permanent_uuid...
I20260812 02:36:54.289615 8318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:54.278586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.60:40291 (local address 127.0.0.1:58878)
0812 02:36:54.278719 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0812 02:36:54.278736 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 02:36:54.278833 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:36:54.279438 (+ 605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:36:54.279441 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:36:54.279644 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:36:54.279779 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:36:54.279788 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:36:54.281024 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:36:54.281030 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:36:54.281936 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:36:54.281951 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:36:54.282571 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:36:54.282576 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:36:54.282679 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:36:54.283312 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:36:54.283338 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:36:54.285110 (+ 1772us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:36:54.287542 (+ 2432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:36:54.287546 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:36:54.287548 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:36:54.287833 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:36:54.288306 (+ 473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:36:54.288312 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:36:54.288315 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:36:54.288433 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:36:54.288953 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:36:54.288959 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:36:54.289093 (+ 134us) client_negotiation.cc:769] Sending connection context
0812 02:36:54.289339 (+ 246us) client_negotiation.cc:240] Negotiation successful
0812 02:36:54.289497 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260812 02:36:54.290282 8405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:54.278832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58878 (local address 127.14.103.60:40291)
0812 02:36:54.279157 (+ 325us) server_negotiation.cc:206] Beginning negotiation
0812 02:36:54.279163 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 02:36:54.279174 (+ 11us) server_negotiation.cc:407] Connection header received
0812 02:36:54.279225 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:36:54.279230 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:36:54.279278 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:36:54.279902 (+ 624us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:36:54.279995 (+ 93us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:54.280856 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:54.282088 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:54.282446 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:54.282887 (+ 441us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:36:54.285271 (+ 2384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:36:54.285275 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:36:54.285277 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:36:54.285300 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:36:54.287143 (+ 1843us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:36:54.287964 (+ 821us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:36:54.287967 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:36:54.287969 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:36:54.288057 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:36:54.288567 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:36:54.288570 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:36:54.288571 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:36:54.288804 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:36:54.289875 (+ 1071us) server_negotiation.cc:1035] Waiting for connection context
0812 02:36:54.289957 (+ 82us) server_negotiation.cc:299] Negotiation successful
0812 02:36:54.290110 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256,"thread_start_us":160,"threads_started":1}
I20260812 02:36:54.292802 8229 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0500951028ea449cb391b558b03cc234: No bootstrap required, opened a new log
I20260812 02:36:54.293661 8312 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9aa8b2189f974391bcd27b61ea5dea46: Bootstrap starting.
I20260812 02:36:54.294567 8406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:54.282102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43736 (local address 127.14.103.62:38107)
0812 02:36:54.283180 (+ 1078us) server_negotiation.cc:206] Beginning negotiation
0812 02:36:54.283192 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 02:36:54.283203 (+ 11us) server_negotiation.cc:407] Connection header received
0812 02:36:54.283255 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:36:54.283260 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:36:54.283308 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:36:54.284255 (+ 947us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:36:54.284376 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:54.285204 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:54.286312 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:54.286700 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:54.287229 (+ 529us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:36:54.290692 (+ 3463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:36:54.290696 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:36:54.290698 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:36:54.290727 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:36:54.292609 (+ 1882us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:36:54.293146 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:36:54.293150 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:36:54.293151 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:36:54.293248 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:36:54.293575 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:36:54.293579 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:36:54.293580 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:36:54.293833 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:36:54.294026 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0812 02:36:54.294237 (+ 211us) server_negotiation.cc:299] Negotiation successful
0812 02:36:54.294383 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1014,"thread_start_us":82,"threads_started":1}
I20260812 02:36:54.296496 8312 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9aa8b2189f974391bcd27b61ea5dea46: Neither blocks nor log segments found. Creating new log.
I20260812 02:36:54.297225 8312 log.cc:824] T 00000000000000000000000000000000 P 9aa8b2189f974391bcd27b61ea5dea46: Log is configured to *not* fsync() on all Append() calls
I20260812 02:36:54.297412 8229 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0500951028ea449cb391b558b03cc234 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0500951028ea449cb391b558b03cc234" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 38107 } } peers { permanent_uuid: "9aa8b2189f974391bcd27b61ea5dea46" member_type: VOTER last_known_addr { host: "127.14.103.61" port: 36977 } } peers { permanent_uuid: "f9ba6a023c574841b6a4960ea035990b" member_type: VOTER last_known_addr { host: "127.14.103.60" port: 40291 } }
I20260812 02:36:54.297645 8229 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0500951028ea449cb391b558b03cc234 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0500951028ea449cb391b558b03cc234, State: Initialized, Role: FOLLOWER
I20260812 02:36:54.297953 8229 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0500951028ea449cb391b558b03cc234 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0500951028ea449cb391b558b03cc234" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 38107 } } peers { permanent_uuid: "9aa8b2189f974391bcd27b61ea5dea46" member_type: VOTER last_known_addr { host: "127.14.103.61" port: 36977 } } peers { permanent_uuid: "f9ba6a023c574841b6a4960ea035990b" member_type: VOTER last_known_addr { host: "127.14.103.60" port: 40291 } }
I20260812 02:36:54.299628 8229 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0500951028ea449cb391b558b03cc234 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:36:54.299947 8312 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9aa8b2189f974391bcd27b61ea5dea46: No bootstrap required, opened a new log
Aug 12 02:36:54 dist-test-slave-2w3w krb5kdc[8137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502214, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.60@KRBTEST.COM for kudu/127.14.103.61@KRBTEST.COM
Aug 12 02:36:54 dist-test-slave-2w3w krb5kdc[8137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502213, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.103.61@KRBTEST.COM
I20260812 02:36:54.304533 8312 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9aa8b2189f974391bcd27b61ea5dea46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0500951028ea449cb391b558b03cc234" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 38107 } } peers { permanent_uuid: "9aa8b2189f974391bcd27b61ea5dea46" member_type: VOTER last_known_addr { host: "127.14.103.61" port: 36977 } } peers { permanent_uuid: "f9ba6a023c574841b6a4960ea035990b" member_type: VOTER last_known_addr { host: "127.14.103.60" port: 40291 } }
I20260812 02:36:54.304795 8312 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9aa8b2189f974391bcd27b61ea5dea46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9aa8b2189f974391bcd27b61ea5dea46, State: Initialized, Role: FOLLOWER
I20260812 02:36:54.305158 8312 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9aa8b2189f974391bcd27b61ea5dea46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0500951028ea449cb391b558b03cc234" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 38107 } } peers { permanent_uuid: "9aa8b2189f974391bcd27b61ea5dea46" member_type: VOTER last_known_addr { host: "127.14.103.61" port: 36977 } } peers { permanent_uuid: "f9ba6a023c574841b6a4960ea035990b" member_type: VOTER last_known_addr { host: "127.14.103.60" port: 40291 } }
W20260812 02:36:54.306244 8422 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0500951028ea449cb391b558b03cc234: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 02:36:54.306370 8422 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 02:36:54.306593 8312 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9aa8b2189f974391bcd27b61ea5dea46 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:36:54.307251 8409 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0500951028ea449cb391b558b03cc234 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0500951028ea449cb391b558b03cc234" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 38107 } } peers { permanent_uuid: "9aa8b2189f974391bcd27b61ea5dea46" member_type: VOTER last_known_addr { host: "127.14.103.61" port: 36977 } } peers { permanent_uuid: "f9ba6a023c574841b6a4960ea035990b" member_type: VOTER last_known_addr { host: "127.14.103.60" port: 40291 } } }
I20260812 02:36:54.307405 8320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:54.290192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58036 (local address 127.14.103.61:36977)
0812 02:36:54.290331 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0812 02:36:54.290335 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:36:54.290350 (+ 15us) server_negotiation.cc:407] Connection header received
0812 02:36:54.290399 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:36:54.290402 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:36:54.290451 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:36:54.290549 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:36:54.298064 (+ 7515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:54.298789 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:54.299960 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:54.300326 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:54.300404 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:36:54.303440 (+ 3036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:36:54.303444 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:36:54.303446 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:36:54.303471 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:36:54.305122 (+ 1651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:36:54.305661 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:36:54.305665 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:36:54.305667 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:36:54.305753 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:36:54.306062 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:36:54.306065 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:36:54.306066 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:36:54.306332 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:36:54.306428 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0812 02:36:54.307104 (+ 676us) server_negotiation.cc:299] Negotiation successful
0812 02:36:54.307232 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260812 02:36:54.307590 8404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:54.290096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.61:36977 (local address 127.0.0.1:58036)
0812 02:36:54.290196 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0812 02:36:54.290213 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 02:36:54.290306 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:36:54.290584 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:36:54.290590 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:36:54.297751 (+ 7161us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:36:54.297876 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:36:54.297885 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:36:54.298944 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:36:54.298947 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:36:54.299817 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:36:54.299829 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:36:54.300557 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:36:54.300560 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:36:54.300639 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:36:54.301093 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:36:54.301107 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:36:54.303281 (+ 2174us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:36:54.305267 (+ 1986us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:36:54.305271 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:36:54.305273 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:36:54.305544 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:36:54.305855 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:36:54.305858 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:36:54.305860 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:36:54.305949 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:36:54.306475 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:36:54.306478 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:36:54.306579 (+ 101us) client_negotiation.cc:769] Sending connection context
0812 02:36:54.306677 (+ 98us) client_negotiation.cc:240] Negotiation successful
0812 02:36:54.306835 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260812 02:36:54.307839 8409 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0500951028ea449cb391b558b03cc234 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 02:36:54.305091 8152 master_runner.cc:186] Error getting master registration for 127.14.103.61:36977: OK, instance_id { permanent_uuid: "9aa8b2189f974391bcd27b61ea5dea46" instance_seqno: 1786502214022307 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 02:36:54.309433 8152 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260812 02:36:54.310828 8399 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.103.60" port: 40291 } has no permanent_uuid. Determining permanent_uuid...
I20260812 02:36:54.312358 8317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:54.296008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58044 (local address 127.14.103.61:36977)
0812 02:36:54.296133 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0812 02:36:54.296137 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:36:54.296151 (+ 14us) server_negotiation.cc:407] Connection header received
0812 02:36:54.296190 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:36:54.296193 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:36:54.296237 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:36:54.296325 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:36:54.297177 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:54.297918 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:54.299133 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:54.299452 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:54.299534 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:36:54.308038 (+ 8504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:36:54.308044 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:36:54.308049 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:36:54.308081 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:36:54.309901 (+ 1820us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:36:54.310466 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:36:54.310469 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:36:54.310471 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:36:54.310559 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:36:54.310878 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:36:54.310882 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:36:54.310883 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:36:54.311128 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:36:54.311327 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0812 02:36:54.312080 (+ 753us) server_negotiation.cc:299] Negotiation successful
0812 02:36:54.312201 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
W20260812 02:36:54.313630 8435 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9aa8b2189f974391bcd27b61ea5dea46: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 02:36:54.313728 8435 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 02:36:54.313942 8424 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9aa8b2189f974391bcd27b61ea5dea46 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0500951028ea449cb391b558b03cc234" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 38107 } } peers { permanent_uuid: "9aa8b2189f974391bcd27b61ea5dea46" member_type: VOTER last_known_addr { host: "127.14.103.61" port: 36977 } } peers { permanent_uuid: "f9ba6a023c574841b6a4960ea035990b" member_type: VOTER last_known_addr { host: "127.14.103.60" port: 40291 } } }
I20260812 02:36:54.314077 8424 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9aa8b2189f974391bcd27b61ea5dea46 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 02:36:54.317582 14748 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 02:36:54.319120 8198 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "0500951028ea449cb391b558b03cc234"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43736
I20260812 02:36:54.319410 8198 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0500951028ea449cb391b558b03cc234 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260812 02:36:54.319523 8198 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0500951028ea449cb391b558b03cc234 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:36:54.321005 8198 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0500951028ea449cb391b558b03cc234 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0500951028ea449cb391b558b03cc234" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 38107 } } peers { permanent_uuid: "9aa8b2189f974391bcd27b61ea5dea46" member_type: VOTER last_known_addr { host: "127.14.103.61" port: 36977 } } peers { permanent_uuid: "f9ba6a023c574841b6a4960ea035990b" member_type: VOTER last_known_addr { host: "127.14.103.60" port: 40291 } }
Aug 12 02:36:54 dist-test-slave-2w3w krb5kdc[8137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786502214, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.103.60@KRBTEST.COM for kudu/127.14.103.60@KRBTEST.COM
I20260812 02:36:54.322814 8285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0500951028ea449cb391b558b03cc234" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "9aa8b2189f974391bcd27b61ea5dea46"
I20260812 02:36:54.323156 8285 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9aa8b2189f974391bcd27b61ea5dea46 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:36:54.324641 8285 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9aa8b2189f974391bcd27b61ea5dea46 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0500951028ea449cb391b558b03cc234 in term 1.
I20260812 02:36:54.325117 8174 leader_election.cc:302] T 00000000000000000000000000000000 P 0500951028ea449cb391b558b03cc234 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0500951028ea449cb391b558b03cc234, 9aa8b2189f974391bcd27b61ea5dea46; no voters:
I20260812 02:36:54.325520 8409 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0500951028ea449cb391b558b03cc234 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:36:54.325763 8409 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0500951028ea449cb391b558b03cc234 [term 1 LEADER]: Becoming Leader. State: Replica: 0500951028ea449cb391b558b03cc234, State: Running, Role: LEADER
I20260812 02:36:54.326149 8409 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0500951028ea449cb391b558b03cc234 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0500951028ea449cb391b558b03cc234" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 38107 } } peers { permanent_uuid: "9aa8b2189f974391bcd27b61ea5dea46" member_type: VOTER last_known_addr { host: "127.14.103.61" port: 36977 } } peers { permanent_uuid: "f9ba6a023c574841b6a4960ea035990b" member_type: VOTER last_known_addr { host: "127.14.103.60" port: 40291 } }
I20260812 02:36:54.326442 8405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:54.311829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58894 (local address 127.14.103.60:40291)
0812 02:36:54.311933 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0812 02:36:54.311937 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:36:54.311949 (+ 12us) server_negotiation.cc:407] Connection header received
0812 02:36:54.311982 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:36:54.311985 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:36:54.312028 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:36:54.312103 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:36:54.312652 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:54.313451 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:54.319059 (+ 5608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:54.319476 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:54.319594 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:36:54.322472 (+ 2878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:36:54.322477 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:36:54.322479 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:36:54.322508 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:36:54.324384 (+ 1876us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:36:54.324940 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:36:54.324943 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:36:54.324945 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:36:54.325021 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:36:54.325363 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:36:54.325367 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:36:54.325369 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:36:54.325697 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:36:54.325792 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0812 02:36:54.326144 (+ 352us) server_negotiation.cc:299] Negotiation successful
0812 02:36:54.326271 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260812 02:36:54.326825 8404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:54.311376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.103.60:40291 (local address 127.0.0.1:58894)
0812 02:36:54.311602 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0812 02:36:54.311616 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 02:36:54.311698 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:36:54.312209 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:36:54.312212 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:36:54.312419 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:36:54.312535 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:36:54.312542 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:36:54.315082 (+ 2540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:36:54.315085 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:36:54.315922 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:36:54.315934 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:36:54.319859 (+ 3925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:36:54.319863 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:36:54.319952 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:36:54.320487 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:36:54.320503 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:36:54.322318 (+ 1815us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:36:54.324507 (+ 2189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:36:54.324510 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:36:54.324512 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:36:54.324833 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:36:54.325108 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:36:54.325111 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:36:54.325113 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:36:54.325191 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:36:54.325884 (+ 693us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:36:54.325887 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:36:54.325981 (+ 94us) client_negotiation.cc:769] Sending connection context
0812 02:36:54.326100 (+ 119us) client_negotiation.cc:240] Negotiation successful
0812 02:36:54.326429 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"mutex_wait_us":173}
I20260812 02:36:54.327368 8198 leader_election.cc:288] T 00000000000000000000000000000000 P 0500951028ea449cb391b558b03cc234 [CANDIDATE]: Term 1 election: Requested vote from peers 9aa8b2189f974391bcd27b61ea5dea46 (127.14.103.61:36977), f9ba6a023c574841b6a4960ea035990b (127.14.103.60:40291)
I20260812 02:36:54.328109 8409 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0500951028ea449cb391b558b03cc234 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0500951028ea449cb391b558b03cc234. Latest consensus state: current_term: 1 leader_uuid: "0500951028ea449cb391b558b03cc234" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0500951028ea449cb391b558b03cc234" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 38107 } } peers { permanent_uuid: "9aa8b2189f974391bcd27b61ea5dea46" member_type: VOTER last_known_addr { host: "127.14.103.61" port: 36977 } } peers { permanent_uuid: "f9ba6a023c574841b6a4960ea035990b" member_type: VOTER last_known_addr { host: "127.14.103.60" port: 40291 } } }
I20260812 02:36:54.328227 8409 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0500951028ea449cb391b558b03cc234 [sys.catalog]: This master's current role is: LEADER
I20260812 02:36:54.330786 8399 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f9ba6a023c574841b6a4960ea035990b: Bootstrap starting.
I20260812 02:36:54.318650 8236 master_runner.cc:186] Error getting master registration for 127.14.103.60:40291: OK, instance_id { permanent_uuid: "f9ba6a023c574841b6a4960ea035990b" instance_seqno: 1786502214258674 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 02:36:54.332993 8236 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260812 02:36:54.333541 8399 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f9ba6a023c574841b6a4960ea035990b: Neither blocks nor log segments found. Creating new log.
I20260812 02:36:54.334381 8399 log.cc:824] T 00000000000000000000000000000000 P f9ba6a023c574841b6a4960ea035990b: Log is configured to *not* fsync() on all Append() calls
I20260812 02:36:54.337294 8440 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 02:36:54.337682 8399 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f9ba6a023c574841b6a4960ea035990b: No bootstrap required, opened a new log
I20260812 02:36:54.338241 8440 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 02:36:54.345196 8399 raft_consensus.cc:348] T 00000000000000000000000000000000 P f9ba6a023c574841b6a4960ea035990b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0500951028ea449cb391b558b03cc234" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 38107 } } peers { permanent_uuid: "9aa8b2189f974391bcd27b61ea5dea46" member_type: VOTER last_known_addr { host: "127.14.103.61" port: 36977 } } peers { permanent_uuid: "f9ba6a023c574841b6a4960ea035990b" member_type: VOTER last_known_addr { host: "127.14.103.60" port: 40291 } }
I20260812 02:36:54.345826 8399 raft_consensus.cc:729] T 00000000000000000000000000000000 P f9ba6a023c574841b6a4960ea035990b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9ba6a023c574841b6a4960ea035990b, State: Initialized, Role: FOLLOWER
I20260812 02:36:54.346422 8399 consensus_queue.cc:260] T 00000000000000000000000000000000 P f9ba6a023c574841b6a4960ea035990b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0500951028ea449cb391b558b03cc234" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 38107 } } peers { permanent_uuid: "9aa8b2189f974391bcd27b61ea5dea46" member_type: VOTER last_known_addr { host: "127.14.103.61" port: 36977 } } peers { permanent_uuid: "f9ba6a023c574841b6a4960ea035990b" member_type: VOTER last_known_addr { host: "127.14.103.60" port: 40291 } }
I20260812 02:36:54.348841 8369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0500951028ea449cb391b558b03cc234" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "f9ba6a023c574841b6a4960ea035990b"
I20260812 02:36:54.349117 8369 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f9ba6a023c574841b6a4960ea035990b [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:36:54.350601 8369 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f9ba6a023c574841b6a4960ea035990b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0500951028ea449cb391b558b03cc234 in term 1.
I20260812 02:36:54.351028 8399 sys_catalog.cc:564] T 00000000000000000000000000000000 P f9ba6a023c574841b6a4960ea035990b [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:36:54.354466 8450 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9ba6a023c574841b6a4960ea035990b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0500951028ea449cb391b558b03cc234" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 38107 } } peers { permanent_uuid: "9aa8b2189f974391bcd27b61ea5dea46" member_type: VOTER last_known_addr { host: "127.14.103.61" port: 36977 } } peers { permanent_uuid: "f9ba6a023c574841b6a4960ea035990b" member_type: VOTER last_known_addr { host: "127.14.103.60" port: 40291 } } }
I20260812 02:36:54.354642 8450 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9ba6a023c574841b6a4960ea035990b [sys.catalog]: This master's current role is: FOLLOWER
I20260812 02:36:54.359851 8406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:54.335287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43738 (local address 127.14.103.62:38107)
0812 02:36:54.335412 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0812 02:36:54.335417 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:36:54.341068 (+ 5651us) server_negotiation.cc:407] Connection header received
0812 02:36:54.343757 (+ 2689us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:36:54.343762 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:36:54.343820 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:36:54.343873 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:36:54.347980 (+ 4107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:54.348785 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:54.352993 (+ 4208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:54.353411 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:54.353493 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:36:54.355988 (+ 2495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:36:54.355992 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:36:54.355995 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:36:54.356026 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:36:54.357724 (+ 1698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:36:54.358249 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:36:54.358254 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:36:54.358256 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:36:54.358353 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:36:54.358662 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:36:54.358665 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:36:54.358667 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:36:54.358922 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:36:54.359100 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0812 02:36:54.359494 (+ 394us) server_negotiation.cc:299] Negotiation successful
0812 02:36:54.359669 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260812 02:36:54.366228 8317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:54.336131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58058 (local address 127.14.103.61:36977)
0812 02:36:54.336245 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0812 02:36:54.336249 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:36:54.340656 (+ 4407us) server_negotiation.cc:407] Connection header received
0812 02:36:54.341636 (+ 980us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:36:54.341640 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:36:54.341691 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:36:54.341741 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:36:54.344880 (+ 3139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:54.345693 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:54.351000 (+ 5307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:54.351326 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:54.351448 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:36:54.362000 (+ 10552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:36:54.362005 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:36:54.362008 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:36:54.362038 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:36:54.364287 (+ 2249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:36:54.364813 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:36:54.364817 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:36:54.364819 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:36:54.364911 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:36:54.365203 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:36:54.365206 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:36:54.365208 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:36:54.365433 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:36:54.365558 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0812 02:36:54.365922 (+ 364us) server_negotiation.cc:299] Negotiation successful
0812 02:36:54.366053 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260812 02:36:54.367424 8405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:54.335716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58904 (local address 127.14.103.60:40291)
0812 02:36:54.335819 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0812 02:36:54.335823 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:36:54.340872 (+ 5049us) server_negotiation.cc:407] Connection header received
0812 02:36:54.341381 (+ 509us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:36:54.341385 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:36:54.341438 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:36:54.341491 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:36:54.346928 (+ 5437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:54.347737 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:54.351615 (+ 3878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:54.351964 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:54.352038 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:36:54.362543 (+ 10505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:36:54.362547 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:36:54.362550 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:36:54.362600 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:36:54.364744 (+ 2144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:36:54.365810 (+ 1066us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:36:54.365814 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:36:54.365816 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:36:54.365907 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:36:54.366197 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:36:54.366200 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:36:54.366202 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:36:54.366424 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:36:54.366553 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0812 02:36:54.367130 (+ 577us) server_negotiation.cc:299] Negotiation successful
0812 02:36:54.367255 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260812 02:36:54.371270 8369 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f9ba6a023c574841b6a4960ea035990b [term 1 FOLLOWER]: Refusing update from remote peer 0500951028ea449cb391b558b03cc234: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:36:54.371496 8285 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9aa8b2189f974391bcd27b61ea5dea46 [term 1 FOLLOWER]: Refusing update from remote peer 0500951028ea449cb391b558b03cc234: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:36:54.372154 8438 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0500951028ea449cb391b558b03cc234 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9aa8b2189f974391bcd27b61ea5dea46" member_type: VOTER last_known_addr { host: "127.14.103.61" port: 36977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260812 02:36:54.372272 8464 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f9ba6a023c574841b6a4960ea035990b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 02:36:54.372349 8464 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 02:36:54.372345 8409 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0500951028ea449cb391b558b03cc234 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9ba6a023c574841b6a4960ea035990b" member_type: VOTER last_known_addr { host: "127.14.103.60" port: 40291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:36:54.377810 8424 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9aa8b2189f974391bcd27b61ea5dea46 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0500951028ea449cb391b558b03cc234. Latest consensus state: current_term: 1 leader_uuid: "0500951028ea449cb391b558b03cc234" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0500951028ea449cb391b558b03cc234" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 38107 } } peers { permanent_uuid: "9aa8b2189f974391bcd27b61ea5dea46" member_type: VOTER last_known_addr { host: "127.14.103.61" port: 36977 } } peers { permanent_uuid: "f9ba6a023c574841b6a4960ea035990b" member_type: VOTER last_known_addr { host: "127.14.103.60" port: 40291 } } }
I20260812 02:36:54.378019 8424 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9aa8b2189f974391bcd27b61ea5dea46 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 02:36:54.377941 8450 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9ba6a023c574841b6a4960ea035990b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0500951028ea449cb391b558b03cc234. Latest consensus state: current_term: 1 leader_uuid: "0500951028ea449cb391b558b03cc234" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0500951028ea449cb391b558b03cc234" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 38107 } } peers { permanent_uuid: "9aa8b2189f974391bcd27b61ea5dea46" member_type: VOTER last_known_addr { host: "127.14.103.61" port: 36977 } } peers { permanent_uuid: "f9ba6a023c574841b6a4960ea035990b" member_type: VOTER last_known_addr { host: "127.14.103.60" port: 40291 } } }
I20260812 02:36:54.378068 8450 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9ba6a023c574841b6a4960ea035990b [sys.catalog]: This master's current role is: FOLLOWER
I20260812 02:36:54.382511 8409 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0500951028ea449cb391b558b03cc234 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0500951028ea449cb391b558b03cc234" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0500951028ea449cb391b558b03cc234" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 38107 } } peers { permanent_uuid: "9aa8b2189f974391bcd27b61ea5dea46" member_type: VOTER last_known_addr { host: "127.14.103.61" port: 36977 } } peers { permanent_uuid: "f9ba6a023c574841b6a4960ea035990b" member_type: VOTER last_known_addr { host: "127.14.103.60" port: 40291 } } }
I20260812 02:36:54.382699 8409 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0500951028ea449cb391b558b03cc234 [sys.catalog]: This master's current role is: LEADER
I20260812 02:36:54.382850 8424 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9aa8b2189f974391bcd27b61ea5dea46 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0500951028ea449cb391b558b03cc234" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0500951028ea449cb391b558b03cc234" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 38107 } } peers { permanent_uuid: "9aa8b2189f974391bcd27b61ea5dea46" member_type: VOTER last_known_addr { host: "127.14.103.61" port: 36977 } } peers { permanent_uuid: "f9ba6a023c574841b6a4960ea035990b" member_type: VOTER last_known_addr { host: "127.14.103.60" port: 40291 } } }
I20260812 02:36:54.383008 8424 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9aa8b2189f974391bcd27b61ea5dea46 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 02:36:54.386143 8467 mvcc.cc:205] Tried to move back new op lower bound from 7317513070057308160 to 7317513069883904000. Current Snapshot: MvccSnapshot[applied={T|T < 7317513070057308160}]
I20260812 02:36:54.386368 8449 mvcc.cc:205] Tried to move back new op lower bound from 7317513070057308160 to 7317513069883904000. Current Snapshot: MvccSnapshot[applied={T|T < 7317513070057308160}]
I20260812 02:36:54.391963 8440 catalog_manager.cc:1347] Generated new cluster ID: 4f7b0fb4bd03411aa2027f8de1fd1a4a
I20260812 02:36:54.392071 8440 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 02:36:54.404807 8438 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0500951028ea449cb391b558b03cc234 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0500951028ea449cb391b558b03cc234" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0500951028ea449cb391b558b03cc234" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 38107 } } peers { permanent_uuid: "9aa8b2189f974391bcd27b61ea5dea46" member_type: VOTER last_known_addr { host: "127.14.103.61" port: 36977 } } peers { permanent_uuid: "f9ba6a023c574841b6a4960ea035990b" member_type: VOTER last_known_addr { host: "127.14.103.60" port: 40291 } } }
I20260812 02:36:54.404982 8438 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0500951028ea449cb391b558b03cc234 [sys.catalog]: This master's current role is: LEADER
I20260812 02:36:54.406258 8450 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9ba6a023c574841b6a4960ea035990b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0500951028ea449cb391b558b03cc234" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0500951028ea449cb391b558b03cc234" member_type: VOTER last_known_addr { host: "127.14.103.62" port: 38107 } } peers { permanent_uuid: "9aa8b2189f974391bcd27b61ea5dea46" member_type: VOTER last_known_addr { host: "127.14.103.61" port: 36977 } } peers { permanent_uuid: "f9ba6a023c574841b6a4960ea035990b" member_type: VOTER last_known_addr { host: "127.14.103.60" port: 40291 } } }
I20260812 02:36:54.406390 8450 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9ba6a023c574841b6a4960ea035990b [sys.catalog]: This master's current role is: FOLLOWER
I20260812 02:36:54.407043 8468 mvcc.cc:205] Tried to move back new op lower bound from 7317513070057308160 to 7317513069883904000. Current Snapshot: MvccSnapshot[applied={T|T < 7317513070057308160}]
I20260812 02:36:54.428375 8440 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 02:36:54.429896 8440 catalog_manager.cc:1504] Loading token signing keys...
I20260812 02:36:54.447472 8440 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0500951028ea449cb391b558b03cc234: Generated new TSK 0
I20260812 02:36:54.448459 8440 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260812 02:36:54.482553 8317 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:36:54.479181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58072 (local address 127.14.103.61:36977)
0812 02:36:54.479317 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0812 02:36:54.479322 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:36:54.481384 (+ 2062us) server_negotiation.cc:407] Connection header received
0812 02:36:54.481601 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:36:54.481605 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:36:54.481659 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:36:54.481714 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:36:54.482407 (+ 693us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58072: BlockingRecv error: recv got EOF from 127.0.0.1:58072 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260812 02:36:54.484611 8406 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:36:54.480774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43742 (local address 127.14.103.62:38107)
0812 02:36:54.480893 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0812 02:36:54.480897 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:36:54.483323 (+ 2426us) server_negotiation.cc:407] Connection header received
0812 02:36:54.483582 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:36:54.483585 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:36:54.483637 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:36:54.483695 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:36:54.484482 (+ 787us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43742: BlockingRecv error: recv got EOF from 127.0.0.1:43742 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260812 02:36:54.485400 8405 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:36:54.480059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58914 (local address 127.14.103.60:40291)
0812 02:36:54.480174 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0812 02:36:54.480178 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:36:54.482986 (+ 2808us) server_negotiation.cc:407] Connection header received
0812 02:36:54.483855 (+ 869us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:36:54.483859 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:36:54.483911 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:36:54.483961 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:36:54.485278 (+ 1317us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58914: BlockingRecv error: recv got EOF from 127.0.0.1:58914 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260812 02:36:54.490208 8317 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:36:54.488535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58086 (local address 127.14.103.61:36977)
0812 02:36:54.488645 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0812 02:36:54.488650 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:36:54.489193 (+ 543us) server_negotiation.cc:407] Connection header received
0812 02:36:54.489440 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:36:54.489443 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:36:54.489522 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:36:54.489576 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:36:54.490099 (+ 523us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58086: BlockingRecv error: recv got EOF from 127.0.0.1:58086 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260812 02:36:54.495280 8317 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:36:54.493605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58100 (local address 127.14.103.61:36977)
0812 02:36:54.493714 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0812 02:36:54.493718 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:36:54.494269 (+ 551us) server_negotiation.cc:407] Connection header received
0812 02:36:54.494463 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:36:54.494466 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:36:54.494522 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:36:54.494574 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:36:54.495160 (+ 586us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58100: BlockingRecv error: recv got EOF from 127.0.0.1:58100 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260812 02:36:54.501454 8317 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:36:54.499823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58106 (local address 127.14.103.61:36977)
0812 02:36:54.499931 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0812 02:36:54.499935 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:36:54.500494 (+ 559us) server_negotiation.cc:407] Connection header received
0812 02:36:54.500690 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:36:54.500694 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:36:54.500760 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:36:54.500813 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:36:54.501336 (+ 523us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58106: BlockingRecv error: recv got EOF from 127.0.0.1:58106 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260812 02:36:54.509680 8317 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:36:54.507910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58118 (local address 127.14.103.61:36977)
0812 02:36:54.508072 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0812 02:36:54.508077 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:36:54.508661 (+ 584us) server_negotiation.cc:407] Connection header received
0812 02:36:54.508885 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:36:54.508889 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:36:54.508948 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:36:54.509002 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:36:54.509551 (+ 549us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58118: BlockingRecv error: recv got EOF from 127.0.0.1:58118 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
W20260812 02:36:54.522660 8317 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:36:54.520784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58130 (local address 127.14.103.61:36977)
0812 02:36:54.520918 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0812 02:36:54.520923 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:36:54.521578 (+ 655us) server_negotiation.cc:407] Connection header received
0812 02:36:54.521815 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:36:54.521820 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:36:54.521876 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:36:54.521932 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:36:54.522531 (+ 599us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58130: BlockingRecv error: recv got EOF from 127.0.0.1:58130 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260812 02:36:54.542809 8317 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:36:54.541403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58136 (local address 127.14.103.61:36977)
0812 02:36:54.541547 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0812 02:36:54.541552 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:36:54.541729 (+ 177us) server_negotiation.cc:407] Connection header received
0812 02:36:54.541891 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:36:54.541894 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:36:54.541941 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:36:54.542016 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:36:54.542641 (+ 625us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58136: BlockingRecv error: recv got EOF from 127.0.0.1:58136 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260812 02:36:54.578548 8317 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:36:54.577214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58148 (local address 127.14.103.61:36977)
0812 02:36:54.577388 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0812 02:36:54.577392 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:36:54.577537 (+ 145us) server_negotiation.cc:407] Connection header received
0812 02:36:54.577664 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:36:54.577666 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:36:54.577719 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:36:54.577798 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:36:54.578392 (+ 594us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58148: BlockingRecv error: recv got EOF from 127.0.0.1:58148 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
W20260812 02:36:54.645965 8317 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:36:54.644673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58150 (local address 127.14.103.61:36977)
0812 02:36:54.644838 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0812 02:36:54.644844 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 02:36:54.645006 (+ 162us) server_negotiation.cc:407] Connection header received
0812 02:36:54.645116 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:36:54.645118 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:36:54.645168 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:36:54.645244 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:36:54.645828 (+ 584us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58150: BlockingRecv error: recv got EOF from 127.0.0.1:58150 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260812 02:36:54.777456 8317 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:36:54.776130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58162 (local address 127.14.103.61:36977)
0812 02:36:54.776317 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0812 02:36:54.776322 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:36:54.776472 (+ 150us) server_negotiation.cc:407] Connection header received
0812 02:36:54.776651 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:36:54.776654 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:36:54.776715 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:36:54.776812 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:36:54.777312 (+ 500us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58162: BlockingRecv error: recv got EOF from 127.0.0.1:58162 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
W20260812 02:36:55.036998 8317 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 02:36:55.035794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58170 (local address 127.14.103.61:36977)
0812 02:36:55.035957 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0812 02:36:55.035962 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:36:55.036089 (+ 127us) server_negotiation.cc:407] Connection header received
0812 02:36:55.036252 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:36:55.036256 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:36:55.036318 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:36:55.036410 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:36:55.036881 (+ 471us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58170: BlockingRecv error: recv got EOF from 127.0.0.1:58170 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
I20260812 02:36:55.315191 8435 catalog_manager.cc:1259] Loaded cluster ID: 4f7b0fb4bd03411aa2027f8de1fd1a4a
I20260812 02:36:55.315292 8435 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 9aa8b2189f974391bcd27b61ea5dea46: loading cluster ID for follower catalog manager: success
I20260812 02:36:55.316933 8435 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 9aa8b2189f974391bcd27b61ea5dea46: acquiring CA information for follower catalog manager: success
I20260812 02:36:55.318076 8435 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 9aa8b2189f974391bcd27b61ea5dea46: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 02:36:55.373735 8464 catalog_manager.cc:1259] Loaded cluster ID: 4f7b0fb4bd03411aa2027f8de1fd1a4a
I20260812 02:36:55.373862 8464 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f9ba6a023c574841b6a4960ea035990b: loading cluster ID for follower catalog manager: success
I20260812 02:36:55.375648 8464 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f9ba6a023c574841b6a4960ea035990b: acquiring CA information for follower catalog manager: success
I20260812 02:36:55.376889 8464 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f9ba6a023c574841b6a4960ea035990b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 02:36:55.555760 8548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:55.551341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58182 (local address 127.14.103.61:36977)
0812 02:36:55.551619 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0812 02:36:55.551623 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:36:55.551752 (+ 129us) server_negotiation.cc:407] Connection header received
0812 02:36:55.551957 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:36:55.551961 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:36:55.552169 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:36:55.552327 (+ 158us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 02:36:55.552666 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:55.553293 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:55.554195 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:55.554419 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:55.554496 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:36:55.554860 (+ 364us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 02:36:55.555393 (+ 533us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 02:36:55.555492 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0812 02:36:55.555579 (+ 87us) server_negotiation.cc:299] Negotiation successful
0812 02:36:55.555630 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":197,"thread_start_us":72,"threads_started":1}
I20260812 02:36:55.562875 8555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:55.559032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58916 (local address 127.14.103.60:40291)
0812 02:36:55.559315 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0812 02:36:55.559319 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:36:55.559337 (+ 18us) server_negotiation.cc:407] Connection header received
0812 02:36:55.559419 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:36:55.559421 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:36:55.559539 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:36:55.559645 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 02:36:55.559879 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:55.560406 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:55.561234 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:55.561473 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:55.561553 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:36:55.561844 (+ 291us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 02:36:55.562282 (+ 438us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 02:36:55.562395 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0812 02:36:55.562615 (+ 220us) server_negotiation.cc:299] Negotiation successful
0812 02:36:55.562674 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":116,"threads_started":1}
I20260812 02:36:55.570538 8555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:55.566065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58918 (local address 127.14.103.60:40291)
0812 02:36:55.566173 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0812 02:36:55.566177 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:36:55.566827 (+ 650us) server_negotiation.cc:407] Connection header received
0812 02:36:55.567045 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:36:55.567048 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:36:55.567103 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:36:55.567176 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 02:36:55.567543 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:55.568287 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:55.569233 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:55.569571 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:55.569675 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:36:55.569963 (+ 288us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 02:36:55.570054 (+ 91us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 02:36:55.570104 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0812 02:36:55.570323 (+ 219us) server_negotiation.cc:299] Negotiation successful
0812 02:36:55.570389 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260812 02:36:55.570840 8548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:55.566020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58186 (local address 127.14.103.61:36977)
0812 02:36:55.566156 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0812 02:36:55.566161 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 02:36:55.566830 (+ 669us) server_negotiation.cc:407] Connection header received
0812 02:36:55.566899 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:36:55.566903 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:36:55.567006 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:36:55.567081 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 02:36:55.567324 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:55.568124 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:55.569360 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:55.569705 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:55.569806 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:36:55.570088 (+ 282us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 02:36:55.570180 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 02:36:55.570249 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0812 02:36:55.570637 (+ 388us) server_negotiation.cc:299] Negotiation successful
0812 02:36:55.570705 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260812 02:36:55.571269 8561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:36:55.566000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43758 (local address 127.14.103.62:38107)
0812 02:36:55.566400 (+ 400us) server_negotiation.cc:206] Beginning negotiation
0812 02:36:55.566404 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 02:36:55.566779 (+ 375us) server_negotiation.cc:407] Connection header received
0812 02:36:55.567035 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:36:55.567039 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:36:55.567232 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:36:55.567350 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 02:36:55.567709 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:55.568347 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:55.569524 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:36:55.569855 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:36:55.569957 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:36:55.570270 (+ 313us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 02:36:55.570665 (+ 395us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 02:36:55.570764 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0812 02:36:55.571050 (+ 286us) server_negotiation.cc:299] Negotiation successful
0812 02:36:55.571123 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":73,"threads_started":1}
I20260812 02:36:55.580454 14748 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbep4M4/build/debug/bin/kudu with pid 8152
I20260812 02:36:55.589818 14748 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbep4M4/build/debug/bin/kudu with pid 8236
I20260812 02:36:55.597922 14748 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbep4M4/build/debug/bin/kudu with pid 8323
2026-08-12T02:36:55Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3333 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6587 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (316840 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST